mongodb - Database is seen in 'show db' even after dropping the database successfully -

we have following configuration of mongodb:(journal enabled) on machines-1 have following servers mongod - 5 config server - 2 mongos - 1 on machines-2 have following servers mongod - 5 config server - 1

we ran dropdatabase command mongos successfully. saw dropped database in output of show dbs command(ran mongos) after dropping database successfully.

we observed show dbs command (ran each mongod) following: 1. dropped database not seen mongod1 , mongod4.

why dropped database seen in output of show dbs command, after dropping database ?

please me resolve issue.

find logs of 00150812 database below:

mongos logs:

line 17937: mon sep 23 22:47:28 [conn17154] dbconfig unserialize: 00150812 { _id: "00150812", partitioned: true, primary: "shard0000" } line 17937: mon sep 23 22:47:28 [conn17154] dbconfig unserialize: 00150812 { _id: "00150812", partitioned: true, primary: "shard0000" } line 17938: mon sep 23 22:47:28 [conn17154] created new distributed lock 00150812.database on 192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000 ( lock timeout : 900000, ping interval : 30000, process : 0 ) line 17939: mon sep 23 22:47:28 [conn17154] chunkmanager: time load chunks 00150812.database: 0ms sequencenumber: 75 version: 28|1 line 17940: mon sep 23 22:47:28 [conn17154] drop database: 00150812 line 17941: mon sep 23 22:47:28 [conn17154] dbconfig::dropdatabase: 00150812 line 17942: mon sep 23 22:47:28 [conn17154] log metadata event: { _id: "css-fu-1-2013-09-23t19:47:28-331", server: "css-fu-1", clientaddr: "n/a", time: new date(1379965648710), what: "dropdatabase.start", ns: "00150812", details: {} } line 17943: mon sep 23 22:47:28 [conn17154]      removed entry config server for: 00150812 line 17944: mon sep 23 22:47:28 [conn17154]      dropping sharded collection: 00150812.database line 17945: mon sep 23 22:47:28 [conn17154] log metadata event: { _id: "css-fu-1-2013-09-23t19:47:28-332", server: "css-fu-1", clientaddr: "n/a", time: new date(1379965648862), what: "dropcollection.start", ns: "00150812.database", details: {} } line 17946: mon sep 23 22:47:28 [conn17154] acquire distributed lock '00150812.database/css-fu-1:35000:1379577748:1804289383: line 17953: { "_id" : "00150812.database", line 17956: mon sep 23 22:47:29 [conn17154] distributed lock '00150812.database/css-fu-1:35000:1379577748:1804289383' acquired, ts : 52409ad0ed8f24e472bdd5d0 line 17957: mon sep 23 22:47:29 [conn17154] chunkmanager::drop : 00150812.database line 17958: mon sep 23 22:47:29 [conn17154] chunkmanager::drop : 00150812.database   locked line 17991: mon sep 23 22:50:50 [conn17154] chunkmanager::drop : 00150812.database   removed shard data line 17992: mon sep 23 22:50:50 [conn17154] chunkmanager::drop : 00150812.database   removed chunk data line 17993: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0000 192.168.11.20:10000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0000", shardhost: "192.168.11.20:10000" } 0x235cad0 line 17993: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0000 192.168.11.20:10000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0000", shardhost: "192.168.11.20:10000" } 0x235cad0 line 17994: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0001 192.168.11.20:15000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0001", shardhost: "192.168.11.20:15000" } 0x7fc388097290 line 17994: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0001 192.168.11.20:15000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0001", shardhost: "192.168.11.20:15000" } 0x7fc388097290 line 17995: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0002 192.168.11.20:20000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0002", shardhost: "192.168.11.20:20000" } 0x7fc388073c70 line 17995: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0002 192.168.11.20:20000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0002", shardhost: "192.168.11.20:20000" } 0x7fc388073c70 line 17996: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0003 192.168.11.20:25000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0003", shardhost: "192.168.11.20:25000" } 0x7fc3880709f0 line 17996: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0003 192.168.11.20:25000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0003", shardhost: "192.168.11.20:25000" } 0x7fc3880709f0 line 17997: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0004 192.168.11.20:30000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0004", shardhost: "192.168.11.20:30000" } 0x7fc3940313c0 line 17997: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0004 192.168.11.20:30000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0004", shardhost: "192.168.11.20:30000" } 0x7fc3940313c0 line 17998: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0005 192.168.11.21:40000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0005", shardhost: "192.168.11.21:40000" } 0x7fc388071760 line 17998: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0005 192.168.11.21:40000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0005", shardhost: "192.168.11.21:40000" } 0x7fc388071760 line 17999: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0006 192.168.11.21:45000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0006", shardhost: "192.168.11.21:45000" } 0x7fc3a40396c0 line 17999: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0006 192.168.11.21:45000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0006", shardhost: "192.168.11.21:45000" } 0x7fc3a40396c0 line 18000: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0007 192.168.11.21:50000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0007", shardhost: "192.168.11.21:50000" } 0x7fc39c0781a0 line 18000: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0007 192.168.11.21:50000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0007", shardhost: "192.168.11.21:50000" } 0x7fc39c0781a0 line 18001: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0008 192.168.11.21:55000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0008", shardhost: "192.168.11.21:55000" } 0x7fc3a81f0ff0 line 18001: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0008 192.168.11.21:55000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0008", shardhost: "192.168.11.21:55000" } 0x7fc3a81f0ff0 line 18002: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0009 192.168.11.21:60000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0009", shardhost: "192.168.11.21:60000" } 0x7fc3940300e0 line 18002: mon sep 23 22:50:50 [conn17154]     setshardversion  shard0009 192.168.11.21:60000  00150812.database  { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0009", shardhost: "192.168.11.21:60000" } 0x7fc3940300e0 line 18007: mon sep 23 22:50:50 [conn17154] chunkmanager::drop : 00150812.database   done line 18008: mon sep 23 22:50:50 [conn17154] log metadata event: { _id: "css-fu-1-2013-09-23t19:50:50-333", server: "css-fu-1", clientaddr: "n/a", time: new date(1379965850345), what: "dropcollection", ns: "00150812.database", details: {} } line 18013: mon sep 23 22:50:50 [conn17154] distributed lock '00150812.database/css-fu-1:35000:1379577748:1804289383' unlocked.  line 18015: mon sep 23 22:50:50 [conn17154]    dbconfig::dropdatabase: 00150812 dropped sharded collections: 1 line 18016: mon sep 23 22:50:50 [conn17154]    dbconfig::dropdatabase: 00150812 dropped sharded collections: 0 line 18051: mon sep 23 22:51:20 [conn17160] couldn't find database [00150812] in config db line 18053: mon sep 23 22:51:20 [conn17160]      put [00150812] on: shard0005:192.168.11.21:40000 line 18054: mon sep 23 22:51:20 [conn17160] drop database: 00150812 line 18055: mon sep 23 22:51:20 [conn17160] dbconfig::dropdatabase: 00150812 line 18056: mon sep 23 22:51:20 [conn17160] log metadata event: { _id: "css-fu-1-2013-09-23t19:51:20-334", server: "css-fu-1", clientaddr: "n/a", time: new date(1379965880999), what: "dropdatabase.start", ns: "00150812", details: {} } line 18057: mon sep 23 22:51:21 [conn17160]      removed entry config server for: 00150812 line 18058: mon sep 23 22:51:21 [conn17160]    dbconfig::dropdatabase: 00150812 dropped sharded collections: 0 line 18060: mon sep 23 22:51:36 [conn17160]      dropped primary db for: 00150812 line 18061: mon sep 23 22:51:36 [conn17160] log metadata event: { _id: "css-fu-1-2013-09-23t19:51:36-335", server: "css-fu-1", clientaddr: "n/a", time: new date(1379965896464), what: "dropdatabase", ns: "00150812", details: {} } line 18062: mon sep 23 22:51:36 [conn17160] couldn't find database [00150812] in config db line 18064: mon sep 23 22:51:36 [conn17160]      put [00150812] on: shard0005:192.168.11.21:40000 

mongod1 logs:

line 19629: mon sep 23 22:47:29 [conn2603] runquery called 00150812.$cmd { drop: "database" } line 19630: mon sep 23 22:47:29 [conn2603] run command 00150812.$cmd { drop: "database" } line 19681: mon sep 23 22:50:43 [conn2603] dropcollection: 00150812.database line 19682: mon sep 23 22:50:44 [conn2603] create collection 00150812.$freelist {} line 19684: mon sep 23 22:50:44 [conn2603] command 00150812.$cmd command: { drop: "database" } ntoreturn:1 reslen:124 708ms line 19719: mon sep 23 22:50:50 [conn2603] runquery called admin.$cmd { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0000", shardhost: "192.168.11.20:10000" } line 19720: mon sep 23 22:50:50 [conn2603] run command admin.$cmd { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0000", shardhost: "192.168.11.20:10000" } line 19721: mon sep 23 22:50:50 [conn2603] command: { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0000", shardhost: "192.168.11.20:10000" } line 19723: mon sep 23 22:50:50 [conn2603] command admin.$cmd command: { setshardversion: "00150812.database", configdb: "192.168.11.20:62000,192.168.11.20:65000,192.168.11.21:38000", version: timestamp 0|0, serverid: objectid('523aaf94ed8f24e472bdd592'), authoritative: true, shard: "shard0000", shardhost: "192.168.11.20:10000" } ntoreturn:1 reslen:57 0ms line 19804: mon sep 23 22:50:50 [conn8756] runquery called 00150812.$cmd { dropdatabase: 1 } line 19805: mon sep 23 22:50:50 [conn8756] run command 00150812.$cmd { dropdatabase: 1 } line 19806: mon sep 23 22:50:50 [conn8756] dropdatabase 00150812 line 19807: mon sep 23 22:50:50 [conn8756] dropdatabase 00150812 line 19837: mon sep 23 22:54:43 [conn8756] remove file /u01/shard1/00150812.ns line 19838: mon sep 23 22:54:43 [conn8756] command 00150812.$cmd command: { dropdatabase: 1 } ntoreturn:1 reslen:59 232912ms 

please me resolve issue.

t&r, jrv