__Server Tools Overview__
* mongostat: quick stat on a running mongod e.g., `mongostat --port 30000`: return mongo stats (such as insert, delete, update and lower memory stats such as vsize (total amount of virtual size used)) every second
* mongorestore, mongodump: import or export dump files in BSON format to collections. these dump files are in BSON format, not human readable. Must authenticate through mongodump in order to dump.

* mongoexport, mongoimport: import or export dump files in JSON format to collections.
    * Mongoexport must convert every document from BSON to JSON. This takes much longer than mongodump, which simply outputs BSON.
    * By default, mongoexport sends output to standard output, but mongodump writes to a file.
    * Mongodump can create a data file and a metadata file, but mongoexport just creates a data file.

Lab: Importing a Dataset
Run a mongoimport command on a MongoDB instance running in the background.
The requirements for this command are:
    * connect to a mongod process running on port 27000
    * import the data from /dataset/products.json
    * import the data to applicationData.products
    * use m103-application-user to authenticate to the database - this user has already been created for you on the admin database with password m103-application-pass

__What is Replication__
* Replication: main multiple copies of your data. A replica set consists of a primary node and a bunch of secondaries. Data are sync from primary to secondary asynchronously. 
* Types of replication (MongoDB only uses statement-based replication)
    * Binary replication: examining what bytes changed in the data file and record these changes into a binary log. The secondaries then receive a copy of the binary log, and syncs changes. Using binary replication is fast, less data needs to be stored to binary log, and sync to secondaries is fast, but it assumes the cluster all nodes are using the same os, same os version, and same mongodb server version.
    * Statement-based replication: after writes is completed on the primary node, the write statement is stored in the oplog, secondaries then sync their oplog with primary's oplog and replay any new operations on their own data. This approach works regardless of the os, os version, and mongodb server version. 
 

__MOngodb Replica sets__
* members in a replica set can be either a primary node, or a secondary node, an arbiter, or hidden nodes. 
* async process could use one of the protocols
    * Protocol version 1: default protocol, this protocol is based out of RAFT protocol. 
    * PV0
* arbiter: a member holds no data, can vote in an election, cannot become primary. Strongly discourage of using arbiters. 
* hidden nodes: can use as delayed nodes, helpful to be resilient to potential corruption. Will not be seen by the application, if you `rs.isMaster()` cannot see this node on the list. A hidden node can never be a primary node. 
* election: when primary node is down, a secondary will be elected to be the primary. This process is call "failover".
* replica set topology: the list of replica set members and their configs define the replica set topology. any topology change will trigger re-election. The maximum number of failovers is determined by when we don't have enough nodes for majority votes.  We can have up to 50 replica set members, but only 7 of those will be voting members.


__Set up a Replica set__
* One conf file for each node in the cluster. 
* connect each node so they are aware of each other. 

Node 1 conf file. Similar to previous config file, adding the following more info
* keyFile: mandates all members of the replica set have to authenticate to each other using this key file
* authorization: enabled: need to authenticate with clients
* Note: Enabling internal authentication in a replica set implicitly enables client authentication. No when keyFile is in place, we don't actually need to explicitly state "authorization:enabled", but in this config, we do because we re-assure we want client authentication. 
* replSetName: define name of the replSet for this node to be member of

Node 2 config file
* node 2 config file differs node 1 config by the dbpath, port, and logpath
* it uses the same keyFile as node 1. Sometimes nodes are running on different machines, this keyFile must be copied to each machine. 

Same for node 3. 

Connect nodes to a replica set
0. start a mongod process use one of conf files `mongod --config node_1.conf`, open mongo shell `mongo --port 27011`
1. initiate a replica set `rs.initiate()`
2. since we have client authentization enabled, we need to create a user and connect as that user. `use admin`, `db.createUser({user: , pwd:, roles:[{role: "root", db: "admin"}]})` (e.g, create a super user at admin db)
    * start the other two mongod processes with their respective configuration files. `mongod --config node_2.conf`, `mongod --config node_3.conf`
3. exit mongo shell `exit` and log back in as that user by specifying the replica set name in the host name `mongo --host "m103-example/m103.mongodb.university:27011" -u "m103-admin" -p "m103-pass" --authenticationDatabase "admin"`
4. add another node e.g., `rs.add("m103.mongodb.university:27012")`, `rs.add("m103.mongodb.university:27013")`: the port these are running on
5. Returns the replica set status from the point of view of the member where the method is run. `rs.status()`
    * "heartbeatIntervalMillis": how often nodes talk to each other
    * "members": list of members in the replica set
6. `rs.isMaster()`: Returns a document that describes the role of the mongod instance.
    * "host" area shows the nodes in the replica set and their ports. 
    * "primary" who is the primary node. 
7. `rs.stepDown`: force the current primary node to safely step down to a secondary and start the election to select the next primary. 
* Note: (Quiz) When connecting to a replica set, the mongo shell will redirect the connection to the primary node. Yes. 

__Replication Configuration__
* a replication configuration document is a BSON document that we manage using a JSON representation that defines the configurations options for all nodes of our replica set
* it is part of replica set topology.
* some important fields in this config file 
    * "_id": name string of the replica set 
    * "version": integer shows the current version of config, even time we change the configure, this number will increment
    * "members": a list of node config
        * "_id": a unique identifier (Final Exam Q4) for each element in the array
        * "host": host of the node, you cannot specify the same host information among multiple members (Final exam Q4).
        * "arbiterOnly": binary
        * "hidden": binary, set to hidden node or not. will not be listed in rs.isMaster()
        * "priority": int between 0-1000 (Final exam Q4), high priority tends to be elected more often, change of priority triggers re-election, priority=0 emits this node to become a primary. 
        * "slaveDelay": int, a replication delay intervals in seconds, this will be applied to the delayed node in the replica set. By setting this field to a positive number, we auto-set this node to be "hidden: true", and "priority: 0". 
    * etc. more fields

__Replication Commands__
* rs.status()
    * gives the most info about the nodes
* rs.isMater(): describe the nodes
* db.serverStatus()['repl']
    * a section from db.serverStatus()
    * similar to the output of rs.isMaster()
* rs.printReplicationInfo(): only returns oplog data reletive to current node
    * only contains timestamps for first and last oplog events, but not the statement entered in the oplog. 

__Local DB__
* local DB contains important info. 
    * oplog.rs: it is a capped collection (has a size limit, by default about 5% of your disk or your specified value) This collection is created after we create a replica set and add nodes to it. 
        * it contains operation statements
        * every node has its own oplog.rs
        * if a secondary node fails to sync with primary node at one operations, it will be placed into a recovery mode, and no further sync
        * has a replication window, old one will be covered by the new operations. Replication window is proportional to the system load. 
        * One operation may result multiple oplog entries. 
    * replSet.election etc files
* don't change any of files in default exisiting in local DB. You can add a new collection in local DB, but whatever your write into that collection, will not be replicated, it stays in local. 

Quiz: Which statement is true?
* You cannot write to the local database. False
* We should drop the oplog.rs collection from time to time to avoid it becoming too big. False. never delete it. 
* The local database will not be replicated. Yes. 
* The local database does not allow the creation of other collections. No, you can create, but whatever you create will stay in local, not be replicated. 
* The oplog.rs collection contains all operations that will be replicated. Yes. 

__Reconfiguring a replica set while it is running__
* add arbiter node `rs.addArb("m103.mongodb.university:27004")`
* `rs.remove("m103:28000")`: remove node by passing in its port
* `cfg = rs.conf()` current conf of the replica set, `cfg.members[3].votes = 0`, `cfg.members[3].hidden = true`, `cfg.members[3].priority = 0`, `rs.reconfig(cfg)`. 
* Which of the following is true about reconfiguring a replica set with rs.reconfig()?
    * It does not require any of the nodes to restarted.
    * It does not require any of the configuration files to be updated.
    * It does not require the entire configuration document. False. When issuing an updated configuration to rs.reconfig(), the entire configuration document is required.

__Reads and Writes on a replica set__
* We can run read commands on secondary only after enabling read commands on a secondary node: `rs.slaveOk()`, then read `use newDB`, `db.new_collection.find()`
* we can only reads on secondaries, never be able to write to a secondary node. 
* we cannot write to primary when majority is not availale (e.g., one secondary node is down, and the rest nodes won't be able to form a majority. )
* Connecting to the replica set will automatically connect to a secondary node. False. By default, MongoDB will always redirect a Replica Set connection to the primary node. In case of Read Preference being defined in the connection string, that determines other behavior, then the connection might be redirected to connect to a Secondary node, only for read workload. Writes are always sent to the Primary node of the Replica Set cluster. To connect to a secondary node, you need to connect through `mongo --host "" --u ""--p ""`


__Failovers and Elections__
* restarting the server on the primary would trigger the re-election
* when we `rs.stepDown()` triggers the re-election
* Nodes with priority 0 cannot be elected primary. Nodes with higher priority are more likely to be elected primary.
* If the current primary cannot reach a majority of the other nodes in the set, even if the primary is still available, it will auto-stepdown to become a secondary. An election cannot take place until most nodes come back to form a majority and the clients cannot connect to the whole replica set because there is no primary. 

Election happens: 
* when current primary is not available: e.g. restart the server
* when the current primary steps down to become a secondary

__Write Concern__
* mongodb write concern supports for all cluster types: standalone, replica set, and sharded set. 
* higher number of replica set member acknowledgement, the replica set has higher durability. 
* Write concern levels: 
    * 0: don't wait for acknowledgement
    * 1: (default): wait for acknowledgement from primary only
    * more than 2: wait for acknowledgement from the primary and one or more secondaries
    * majority: wait for acknowledgement from a majority of replica set (ceil(total nodes/2)) only.
* write concer options: 
    * wtimeout: the time to wait for the requested write concern before making the operation. Hitting a wtimeout error doesn't mean the write operation itself has failed. 
    * j: boolean, requires the node to commit the write operation to the journal before returning an acknowledgement. (j=false,receive data in memory before reporting the acknowledgement, j=true, not only the data has been received, it has been written to a on-disk journal.)

__Read Concern__
Read concern levels: 
* (latest and fast) local: returns the most recent data in the cluster, any data freshly writes to primary is qualified for local read concern. There is no guarantee the data is safe during a failover event. Local is the default read concern for primary node. 
* (latest and fast) available 
    * for replica set: same as local
        * default read concern for secondary nodes
    * for sharded cluster: has special behavior
* (fast reads and safe) majority: returns data that has acknowledged by and writted into majority nodes, 
* (latest and safe, slower than other read concern, single document reads only) linearizable: like majority, also returns data when it is majority committed. It beyonds majority by it has read you own write functionality. 

__Read Preference__
`db.<collection>.find({}).readPref("secondaryPreferred")`, allows client app to read from whatever node they prefer. 

Read Modes: 
* primary (default): read from primary only
* primaryPreferred: read from primary if available, if primary is down, read from one secondary, possible to return stale data
* secondary: from secondary,  possible to return stale data
* secondaryPreferred: read from secondary if available, if secondary is down or not exist, read from primary, h possible to return stale data
* nearest: read from any node that has the least latency to the client app, regardless of node type, possible to return stale data

Lab: Read preferences: 
Consider a 3-node replica set that experiences a network outage. Two of the three nodes were unreachable during the outage, leaving one node remaining. Which of these readPreferences will allow you to read data from this node?
- secondaryPreferred, primaryPreferred, secondary, nearest
- Reason: when two nodes are down in a 3-node replica set, the remaining node doesn't have majority votes support, thus is stepped down to become a secondary. 

__Lab: Writes with Failovers__
Evaluate the effect of using a write concern with a replica set where one node has failed.

Consider a 3-node replica set with only 2 healthy nodes, that receives the following insert() operation:

* When a writeConcernError occurs, the document is still written to the healthy nodes. This is correct. The WriteResult object simply tells us whether the writeConcern was successful or not - it will not undo successful writes from any of the nodes.
* The unhealthy node will have the inserted document when it is brought back online. This is correct. When the unhealthy node comes back online, it rejoins the replica set and its oplog is compared with the other nodes' oplogs. Any missing operations will be replayed on the newly healthy node.
* w: "majority" would also cause this write operation to return with an error. False. False. w: "majority" requests acknowledgement that a majority of nodes in a replica set have registered the write. In a three-node replica set, only two nodes are required for a majority, so the two healthy nodes are sufficient to satisfy this writeConcern.
The write operation will always return with an error, even if wtimeout is not specified.
* The write operation will always return with an error, even if wtimeout is not specified. False. If wtimeout is not specified, the write operation will be retried for an indefinite amount of time until the writeConcern is successful. If the writeConcern is impossible, like in this example, it may never return anything to the client.

__What is sharding__
* to distribute our large amount of data across cluster, having more resources to store the data. 
* mongos: clients connects to mongos, and mongos routes queries to shards, can have multiple mongos
* config server: contains metadata about the each shard, have to ensure data are relatively evenly distributted to shards. 
* shards: store distributed collections


__When to Shard__
* if it is still economically feasible to vertical scaling, if not, consider sharding
* government regulations require data to be located in a specific geography.
* our organization outgrows the most powerful servers available, limiting our vertical scaling options.
* we are holding more than 5TB per server and operational costs increase dramatically.
* if you rely heavily on aggregation pipeline commands, and processing time becomes slower over time
* Our server disks are full. (Quiz question: Incorrect reason for sharding. Maxing out the capacity of our disks is not a reason for sharding. Scaling up might make more sense than to add complexity to our system.)

__Set up a sharded cluster__
1. write conf file for each shard
    * "sharding": indicates it is a shard node 

2. `mongod -f <each conf>` to start all shard server
3. go to one server `mongo --port 26001`
4. rs.initiate()
5. create a user `use admin`, create super user `db.createUser({user: "m103-admin", pwd: "m103-pass", roles:[{role: "root", db: "admin}]})`, `db.auth("m103-admin", "m103-pass")`
6. adding node to the rs. `rs.add("192.168.103.100:26002")`
7. specify mongos conf file
    * The mongos configuration file doesn't need to have a dbpath. Quiz
    * The mongos configuration file needs to specify the config servers. Quiz

8. start the mongos server `mongos -f mongos.conf`
9. connects to mongos `mongo --port <mongos port: e.g., 26000> --username m103-admin --password m103-pass --authenticationDatabase admin`
10. check sharding status `sh.status()`
    * "shards": number of shards connected
    * "active mongoes": mongos connected
11. add shard in the cluster. `sh.addShard("<shard port>")`


__Config DB__
* generally never need to write to this db
* `db.chunks.find().pretty()`: how this chunk is constructed, what cutoff it uses, which shard node it is at

__Shard key__
* shard key fields must exist in every document in the collection and every new document inserted. 
* ideally, a shard key shall support the majority of queries you run on the collection, that way, the majority of your read operations can be targeted to a single shard, o.w., it needs to go to each shard to match a query. (Quiz)Shard keys are used to route queries to specific shards. This answer is correct. Shard keys are used to organize documents into chunks and shards, and mongos can use this to route queries.
* shard key fields must be indexed
    * indexes must exist first before your can select the indexed fields for your shard key
* the shard key values are mutable, even though the shard key is immutable. Once you shard a collection, you cannot unshard
* how to shard? 
    1. `sh.enableSharding("m103")` to enable sharding for the specified database
    2. create an index `db.products.createIndex( { "sku": 1 } )`
    3. use `sh.shardCollection( "m103.products", { "sku": 1 } )`
    4. check sharding status `sh.status()`

__Picking a good Shard key__
1. cardinality: high cardinality= many possible unique shard key values
2. frequency of a shard key: high freq=low repetition of a given unique shard key value
3. type of change: avoid shard keys to change monotonically (increase/decrease): o/w all data will end up being in the same chunk in the same shard, so sharding on "_id" field is not a good idea
4. if possible, choose a shard key that supports read isolation e.g., fit for most of your queries on the collection so queries can only look for matches in a particular shard rather than looking over all shards. 
* a compound index could be a shard key
* cannot unshard a collection once sharded
* cannot update the shard key of a standard collection
* cannot update the values of the shard key for any document in the sharded collection. 
* test your shard keys in a staging env first before sharding in production envs. 
* How to set a shard key: `sh.shardCollection( "database.collection", { <shard key> } )`


__Hashed shard keys__
* a shard key whose underlying field is hashed. 
* with a hashed shard key, mongodb first hash the field value e.g., 3, then use this hashed value to decide which chunk to place the document in 
* the document fields values is not hashed, instead, the underlying index in hashed and used to partition your data so you end up with a more even distribution acorss the sharded cluster. 
* When do you want to sue the hashed shard key?
    * hashed shard keys provide more even distribution partition of monotonically-changing shard keys. 
* Cons of hashed shard key: 
    * queries on ranges of shard key values need to visit multiple shard nodes now. 
    * cannot support geographically isolated read operations using zoned sharding
    * hashed index must be on a single non-array field
    * hashed index doesn't support sorting
* How to use it 
    1. `sh.enableSharding("<database>")`
    2. `db.collection.createIndex({"<field>": "hashed"})` to create the index for your shard key fields
    3. use `sh.shardCollection("<database>.<collection>", {<shard key field>: "hashed"})` to shard the collection using hashed shard key. 
* Quiz: Even distribution of a monotonically changing shard key field in a compound index. This is incorrect. You cannot create a hashed index on multiple fields.

__Balancing__
* Mongodb balancer determines which shard has most chunks, and move chunks across shards in the sharded cluster in an attempt to achieve even data distribution. 
* `sh.startBalancer(timeout, interval)`, 
    * `sh.stopBalancer(timeout, interval)`: if stop the balancer in the middle of balancing, it will stop until this round of balancing completes.  
    * `sh.setBalancerState(boolean)`
* balancer runs on primary member of config server replica set. Only the Primary of the Config Server Replica Set(CSRS) is responsible for running the balancer process. Neither the secondary CSRS members nor any of the shards are responsible for the balancer process.
* balancer is an automatic process. 
* a given shard cannot participate in more than one migration at a time. floor(n/2) of chunks cannot be migrated in a balancer round. 

__Queries in the sharded cluster__
* mongos handles all queries getting from client app in the cluster. After receiving the query, mongos first decides a list of shards to target a query, mongos merges the results from each shard and return results to client app. 
* queries of sort(): the mongos pushes the sort to each shard, and merge-sorts the results
* queries of limit(): the mongos passes the limit to each targeted shard, then re-applies the limit to the merged set of results. 
* queries of skil(): the mongos performs the skip on the merged set of results, and doesn't push anything down to the shard level.  

__Chunks__
* minKey = -inf, maxKey=inf, left boundary inclusive, right boundary exclusive
* chunk can only live at one shard node at a time. 

__Targeted queries vs scatter gather__
* targeted queries: mongos can use shard key to route query to a specific shard
    * targeted queries require the shard key in the query    
* scatter gather: otherwise 
    * ranged queries on the shard key may still require targeting every shard in the cluster
    * without the shard key, the mongos must perform a scatter-gather query

__Lab: Shard a Collection__
1. Use mongoimport to import the data in /dataset/products.json:
    * import this dataset onto m103.products
    * use mongos as the target for mongoimport - you can find the configuration details for this process in mongos.conf
    * authenticate to mongos as m103-admin (with password m103-pass)
`mongoimport --port 26000 /dataset/products.json --username m103-admin --password m103-pass --authenticationDatabase admin --db m103 --collection products`
2. connect to database using mongo `mongo --port 26000 --username m103-admin --password m103-pass --authenticationDatabase admin`
3. Enable sharding on the m103 database.`sh.enableSharding("m103")`Two shards have already been added to your cluster, shard1 and shard2. check `sh.status()`
4. `use m103` and create index `db.products.createIndex({regularPrice: 1})`
4. `sh.shardCollection( "m103.products", {sku:1 } )`



__Final Exam Q4__
根据题意可知“_id”：3的节点会延迟1个小时，不参与投票，优先度为0（不会成为主节点）

It serves as a reference to perform analytics on how data is changing over time.
它可以作为数据随时间变化的分析参考。错误。 
答案显示这选项错误，但我认为也可以说得通，因为可以通过对比3号节点和其他节点的数据来了解数据的变化情况。

It serves to service reads and writes for people in the same grographic region as the host machine.
它用于为与主机相同的区域内的人员提供读取和写入服务。错误。 
3号节点的优先度为0，是不能提供读写服务的。

It serves as a hidden secondary available to use for non-critical analysis operations.
它可以作为一个隐藏的二级可用于非关键分析操作。错误。 
3号节点有1小时延时，他的数据并不是最新的，无法用来做分析。

It serves as a “hot” backup of data in case of accidental data loss on the other members,like a DBA accidentally dropping the database.
在其他成员意外丢失数据的情况下，它可以作为数据的“热”备份，比如说DBA意外删除了数据库。正确。 
3号节点延时1小时，所以数据库删除命令在它上面也会延时一小时执行。

所以答案为：

It serves as a “hot” backup of data in case of accidental data loss on the other members,like a DBA accidentally dropping the database.

__Final Exam Q6__
Given the following shard key: `{ "country": 1, "_id": 1 }`, Which of the following queries will be routed (targeted)? Remember that queries may be routed to more than one shard.
* db.customers.find({"country": "Norway", "_id": 54}) yes. 
* db.customers.find({"_id": 914, "country": "Sweden"}) yes
* db.customers.find({"country": { $gte: "Portugal", $lte: "Spain" }}) Yes
* db.customers.find({"_id": 455}) No. 

## M201
__Hardware considerations and configs__
* memory/RAM: aggregation, index traversing, write operations, query , connections heavily depend on RAM. More memory, more performant. 
* CPU: to storage engine and concurrency model. MongoDB tries to use all cpu cores to respond incoming requests. 
* Disk RAID architecture can impact the performance of your MongoDB deployment.
* CPU availability impacts the performance of MongoDB, especially if using the WiredTiger storage engine.
___Understand explain()__
Three types: 
1.  `db.people.explain()` or `db.people.explain("queryPlanner")` used to validate the efficiency of your command before execute it. 

* `db.people.explain("executionStats")`: will execute the query and return a different stats about the query. 
* `db.people.explain("allPlansExecution")`: will execute the query. Will return both winning plans and other plans we rejected during performing queries. E.g., when you have multiple indices, and for your query, you might have differnt plans, each for which index to use first in your query. One of them is winning plan, while others are rejected plans, all plans will display if choose allPlansExecution. 

Note 
* in the `inputStage` of the output, stages are nested, the inner most is the first stage to be executed, and outer most is the last. So users can understand the execution stage order. 
* `memUsage` and `memoLimit`, if memory used is bigger than memory limit, the system will throw an exception instead of returning results.
* `stage` : "SORT" tells us the index was not use for the sort and a sort had to done, so it had to be done in memory.
* `executionTime`

Each stage passes its results (i.e. documents or index keys) to the parent node. The leaf nodes access the collection or the indices. The internal nodes manipulate the documents or the index keys that result from the child nodes. The root node is the final stage from which MongoDB derives the result set.

Stages are descriptive of the operation; e.g.

* COLLSCAN for a collection scan
* IXSCAN for scanning index keys
* FETCH for retrieving documents
* SHARD_MERGE for merging results from shards
* SHARDING_FILTER for filtering out orphan documents from shards

SORT Stage: 
    - If MongoDB cannot use an index or indexes to obtain the sort order, the results include a SORT stage indicating a blocking sort operation. 
    - There is not a SORT stage because the documents are already extracted using the index, and so they are already sorted.


__Understanding Explain for Sharded Clusters__
* For sharded clusters, the last stage will be called `SHARD_MERGE` to merge all shards and `shards` field for each standalone shard. 
* Quiz question: With the output of an explain command, what can you deduce? 

{
  "_id": ObjectId,
  "product_name": String,
  "product_id": String
}
with index `{ product_id: 1 }`
Which of the following queries will use the given index to perform the sorting of the returned documents?
- db.products.find({}).sort({ product_id: 1 })
- db.products.find({ product_name: 'Soap' }).sort({ product_id: 1 })

{ "first_name": 1, "address.state": -1, "address.city": -1, "ssn": 1 }

Which of the following queries are able to use it for both filtering and sorting?
- db.people.find({ "first_name": "Jessica", "address.state": { $lt: "S"} }).sort({ "address.state": 1 }) Yes, query and sorting on the same field is okay
- db.people.find({ "address.state": "South Dakota", "first_name": "Jessica" }).sort({ "address.city": -1 }) Yes, the order of predicates doesn't matter
- db.people.find({ "address.city": "West Cindy" }).sort({ "address.city": -1 }) No, address.city is not a prefix
- db.people.find({ "first_name": { $gt: "J" } }).sort({ "address.city": -1 }) No
- db.people.find({ "first_name": "Jessica" }).sort({ "address.state": 1, "address.city": 1 }) Yes

Which of the following statements apply to index resource allocation?
- For the fastest processing, we should ensure that our indexes fit entirely in RAM. yes. 
- Index information does not need to completely allocated in RAM since MongoDB only uses the right-end-side to the index b-tree, regardless of the queries that use index. No. Index information does not need to completely allocated in RAM in the case when your fields grow monotonically, like date, increment IDs. Index is like a b-tree, for monotonically grow data, your tree will grow on right-end-side. Because if we only need to query on the most recent data, then the amount of index that actually needs to be in RAM is always going to be the right-side of your index. 
- Indexes are not required to be entirely placed in RAM, however performance will be affected by constant disk access to retrieve index information. Yes. 

__More Examples on Explain()__
* More examples: https://www.percona.com/blog/2018/09/06/mongodb-investigate-queries-with-explain-index-usage-part-2/

Case 1: index on `borough: 1`, query on the same index `exp.find( { borough: "Manhattan"} )`: directly fetch

Case 2: index on `db.restaurants.createIndex({cuisine:1})`, query on more fields `exp.find({$and: [  {"cuisine" : {$eq :"Italian"}},  {"grades.score" : {$gt : 50}}  ]  })`: more fetch after IDXSCAN

Case 3: index on `db.restaurants.createIndex({cuisine:1})`, query and sort on `exp.find( {"cuisine" : {$ne : "American "}, "grades.grade" :"A", "borough": "Brooklyn"}).sort({"cuisine":-1})`: first IXSCAN, then fetch, but no SORT, because the documents are already extracted using the index, and so they are already sorted

Case 4: index on `db.restaurants.createIndex({"borough" : 1})`, query and sort on `exp.find( {"cuisine" : {$ne : "American "}, "grades.grade" :"A", "borough": "Brooklyn"}).sort({"name":1})`: first IXSCAN, then fetch, but SORT needed, because the index used to fetch the document is borough_1, and MongoDB cannot use an index or indexes to obtain the sort order.

__Optimizing your CRUD Operations__
* index selectivity: minimize no. of index keys we examine
* equality, sort, range rule: useful when building an index, select an index that first satisfy equality query, then sort, lastly range query. E.g., query  `.find({"address.state": {"$ge": 50}, "name": "WA"}).sort("stars": -1)`, choose index `.createIndex({"name": 1, "stars": 1, "address.state": 1})`, rather than `{"name": 1, "address.state": 1, "stars": 1}`, because later uses in-memory sort as query predicates contains a range condition.
* tradeoff between the above two 


__Covered Queries__
* what are covered queries? very performant, satisfied entirely by index keys, 0 docs need to be examined, and MongoDB can both match the query and return the results only using index.  
* Querying only the index is much faster than querying on docs outside of index because index keys are typically smaller than docs they catalog, and **index keys are typically avaible in RAM**. (refer to Resource Allocation for Index section above). 
* How to use it? `.find({"name": "WA", "rates": 5.0}, {"_id": 0, "name": 1, "cuisine": 1, "stars": 1})`, first create index `{"name": 1, "rates": 1}`. Second, in query, remember to add a projection step to eliminate any fields that are not part of the index. 
     * query can include range conditions. 
     * it is a covered query only if all fields in the query are part of the index and all fields turned into results are the same index. 
     * Indication of a covered query: in explain() output, "totalDocsExamined": 0
* Cannot cover a query if 
    * any of the indexed fields are arrays/embedded docs
    * when run against a mongos if the index doesn't contain the shard key. 

__Regex Performance__
* suggest to create a index on the text field (don't have to be text index, could just be a normal index), then apply regex query. 
* make use of carrot regex: `db.products.createIndex({ productName: 1 }), db.products.find({ productName: /^Craftsman/ })`, query everything starting with Craftsman.  The use of the caret at the beginning reduces the number of keys examined. So the query will not need to look at all index keys.

__Aggregation Performance__
Two kinds of aggregation queries: 
* "realtime" processing (query performance is more important, provide data for applications)
* batch processing (query performance is less important, provide data for analytics)

Index Usage: 
* when the server encounters a stage that is not able to use the index, all of the following stages will no longer be able to use the index either. so put stages that can use index at the front of the pipeline. 
* use `$limit` and `$sort` close to each other and at the front of the pipeline in order for mongodb to utilize the top-k sorting alg

Memory Constraints: 
* results are subject to 16MB document limit (use $limit and $project to reduce size)
* 100M of RAM per stage 
    * ensure your largest stage is able to utilize index because index is generally smaller than documents they reference. 
    * use index in sorting. 
    * Last resort measure (万不得已时): use "allowDiskUse" to spill to disk rather than doing everything in RAM, accessing data from disk is slow, so using disk will experience serious performance degration. "allowDiskUse" doesn't support in "$graphLookup". 


## Connect to Atlas and access DB from Mongo Shell
__What is Atlas__
* your db in the cloud
* MongoDB is used at the core of Atlas for data storage and retrieval

__Using MongoDB Atlas__
1. Create a cluster
2. Load sample dataset
3. setup your IP and credentials and obtain the connection URI

__Upload data to Atlas__
* using mongoimport. Install mongoimport, download [MongoDB Database Tools](https://www.mongodb.com/try/download/database-tools) to your local computer
* Open terminal, type `<local path to mongoimport> --drop -c <collection name> --uri mongodb+srv://<cluster username>:<cluster pwd>@sandbox.sibrl.mongodb.net/<db name> <local path to data file e.g., json file>`: 

__Clusters__
* users can create clusters on Atlas
* clusters are a group of servers that stores your data

__Replica set__
* a few connected MongoDB instances that store the same data, where an instance is a single machine locally or in the cloud, running a certain software. 
* single cluster in Atlas is automatically configured as a replica set. 

__JSON vs BSON__
* JSON is text-based format and text parsing is very slow and it is space-consuming, only support string, Boolean, Number and array， it is human and machine readable. 
* BSON is optimzed for speed and space, and supports more data types such as Date and raw binary, and it is machine readable. 
* mongoexport to export BSON objects stored in MongoDB to a JSON/csv file. 
* mongoimport to import JSON objects to MongoDB
* mongodrop to drop db
* mongodump exports data in its raw BSON form.
* mongorestore imports data from a mongodump created BSON format.

__Create and Manipulating Documents__
* 1. connect to cluster
* 2. `use <db name>` e.g., `use sample_training`
* 3. `db.<collection name>.findOne()`: check an example exsiting in db, `db.<collection name>.find()`, `db.<collection name>.find().count()`
* 4. Insertion: `db.<collection name>.insert(<document>)`： insert one doc, `db.<coll name>.insert([<doc1>, <doc2>,etc])`: insert multiple docs, `db.<coll name>.insert(, {"ordered": false})`: disable the default ordered insert. when "ordered=True": insertion stops when encounter errors and will prevent the operation from reaching the other documents.
* 5. `show collections`: view available collections in current db, `show dbs`: When all collections are dropped from a database, the database no longer appears in the list of databases when you run show dbs.
* Update and update operators: `db.<collection name>.updateOne({<specify query}, {<specify updates})`: update the first one matches the query, `db.<collection name>.update(<document>)`: update all docs matches the query
    * `$inc: {"<field>": <inc value>, <field 2>: <value2>}`
    * `$set: {<field 1>: <set field to this value>, <field 2>: <value2>}`: if the field doesn't exist, it will add this specified field
    * `$push : {"<array field>: {< field 1>: <set field to this value>, < field 2>: <value2>}, "<arr field 2>: <field1>: <value1>}`: add an element to an array field
* Delete: `deleteOne()`, `deleteMany()`, `.drop()`: Drop the inspection collection.


__Schema validation__
* MongoDB has schema validation that allows users to enforce document structure. 

__Comparison Operators__
* `{<field>: {$<comparison operator>: <value>}}`
* e.g., `db.trips.find({ "tripduration": { "$lte" : 70 },
                "usertype": { "$ne": "Subscriber" } }).pretty()`

__Logical operators__
* `$and, $or, $nor, : {$<operator: [{statement 1}, {statement 2}]>}`
* `$not: {$not: {statement}}`
* Explicitly use $and when nedd to include the same operator more than once in a query
* could use implicit $and for several queries one the same field. e.g., `{"pop": {"$le": 10, "$ge": 5}}`
* e.g., 

* $and is default expression, and can be used implicitly and explicitly. e.g., Using the sample_airbnb.listingsAndReviews collection find out how many documents have the "property_type" "House", and include "Changing table" as one of the "amenities"?

__$expr__
* allows for more complex queries and for comparing fields within a document

* $: either an operator, or meaning $ for values for field, the second usage can only be used within "$expr" clauses. 

* Incorrect example. This syntax is confusing for MQL, it is trying to look at every number of employees field and compare it to some specific numeric value, but that numeric value is not specified, it is pointing to the value of "founded_year" field but how do we know that we're looking at the same document when we call the "number_of_employees" field? There is no way to tell, so this query will return a zero. 


__Array Operator__
* `$push`: allow us to add an element to an array, and turns a field into an array field if it was previously a diff type

* `{<array field>: {$all: <number>} `: returns a cursor with all docs in which the specified array field contains all the given elements regardless of their order in the array. 
* `{<array field>: {$size: <number>}`: returns a cursor with all docs where the specified array field is exactly the given length.  
* `{<array field>: {"$elemMatch": {<field in array item object>: <value>}}}`: matches docs that contain an array field with at least one element that matches the specified query criteria. i.e., array elements are objects with multiple fields. 
* `db.listingsAndReviews.find({ "amenities.0": "Internet" },{ "name": 1, "address": 1 })`: return the names and addresses of all listings from the sample_airbnb.listingsAndReviews collection where the first amenity in the list is "Internet"?


__Projection__
* `.find({query}, {projection})`: 1. include the field, 0: exclude the field
* use only 1s or 0s otherwise will cause an error "Projection cannot have a mix of inclusion and exclusion"


__MQL vs Aggregation syntax__

__Aggregation Framework__
* aggregation does more than query operations (MQL), it can also be used to compute, re-shape and re-organize data. 
* $group: 

 __Cursor Methods__
 * sort(), limit(), pretty(), count(): it doesn't deal with data in the db. It is applied to results lived in the cursor. so we use the cursor methods after find(). 

* if use both sort and limit, MongoDB assumes always sort first, then limit, regardless of the order in which you type these.       

__Indexing__

* Index Overhead: 
    * don't create unnecessary indexes in a collection. When there is a new doc coming into the collection/being removed/being updated, one or more b-trees need to be reblanced and indexes need to be updated 
* Create an Index: 

    * we can only use index for filtering and sorting if **keys in our predicates are equality conditions**
        * e.g., index on {"name", "address.zip", "ssn"}, but `.find("name": "Judy", "address.zip": {"$gte": 100}).sort({"ssn": 1})`, sort will not use index because "address.zip" uses range query rather than equality query. 
    
    
    
    * Partial indexes: create indexes on some documents in the collection. 
        * motivation: reduce index keys to reduce memory cost
        * e.g., `.createIndex({"address.city": 1, "cuisine": 1}, {partialFilterExpression: {"stars": {"$gte": 3.5}}})`
        * sparse index is a special case of partial indexes. It only index on documents where the field exists, rather than creating an index key for the null value. `.createIndex({'stars': 1}, {sparse: true})`
        * In order to use the partial indexes, the query needs to be guaranteed to match the subset of documents specified by the filter expression in the partial indexes. For example, for the partial index defined above, query like `.find({"address.city": "Vancouver", "cuisine": "Korean"})` will not use IXSCAN. Query like `.find({"address.city": "Vancouver", "cuisine": "Korean", "stars": {"$gte": 4}})` will use IXSCAN. 
        * "_id" indexes cannot be partial indexes. 
        * Shard key indexes cannot be partial indexes. 
        * Partial indexes support a uniqueness constraint. However, uniqueness will be limited to the keys covered by the partial filter expression. Uniqueness constraint means `createIndex( { "email" : 1 }, { unique : true } )`. 
        * Partial indexes support compound indexes.
    

* Indexing Usage: 
    * if query is matching docs on multiple fields, but the database only has a single field index for one of them, the database will first filter using the index and it will only look at these documents and fetch only the ones that match the other predicates. This increases the performance. 
* Examine the query process: The "explain" method
    * what does "explain" method do? To examize if 
        * your query using the index
        * your query using an index to provide a sort
        * your query using an index to provide the projetion
        * how selective is your index
        * which part of your plan is the most expensive. 
    * `db.collection.find().explain("executionStats")`/`.explain("queryPlanner")`/`.explain("allPlansExecution")`
    * look at "queryPlanner" field of output
    * this field has sub-fields like "winningPlan.stage". When "winningPlan.stage"=="COLLSCAN", means that it scans through every document in the collection. When "winningPlan.stage"=="IXSCAN", means that it uses indexes. 
    * sub-field "executionStats":   
        * "totalDocsExamined" for this query, would like "totalDocsExamined" close to "nReturned"
        * "totalKeysExamined": total indexes used for this query, would like "totalKeysExamined" close to "nReturned"
        * if there is no "SORT" stage because the documents are already extracted using the index, and so they are already sorted, otherwise, it doesn't sort using indexes, it is an in-memory sort. 
        * outputs in a sharded cluster have "sharded_merge" stage with execution plans in each of the shards. 
    * in "explain" method, you will NOT see all the available indexes for this collection. you will be able to see the ones considered by the other plans that were rejected, but this is prossibly only a subset of all indexes. 
* Sort with indexes. Sorting is time-consuming, so it's better to use the right indexes for all queries that you sort. 
    * If the index is created with an ascending order, and your query and sort by a descending order, it will still use the index, and your will see "IXSCAN" with direction "backward". 


__CRUD Optimization__
* Optimize CRUD operations: 
    * best practice to order index keys: when create an index, first put key field that matches equality conditions in query, then sorting key field, lastly key field that specified range inequality in query. 
    * performance tradeoff: sometimes to be less selective to prevent an in-memory sort
* Covered Queries
    * what are convered queries? queries are satisfied entirely by index keys, thus 0 docs need to be examined.
    * cannot cover a query if 
        * any of the indexed fields are arrays
        * any of the indexed fields are embedded documents
        * when run against a mongos if the index does not contain the shard key.  
* Regex Performance
    * how to utilize index on regex?
    * When it is `.find({"field 1": /^craftman/})`, (match field 1 starting with "craftman"), for this case, if there is an index of field 1, server will use IXSCAN rather than COLLSCAN. 
* Aggregation Performance
    * when encounter a stage in the pipeline that cannot use the index, all the following stages are not able to use indexes as well. Transforming data in a pipeline stage prevents us from using indexes in the stages that follow. That's why it's important to put all your index using operators at the front of your pipelines!
    * other concepts are memtioned in the Aggregation section below. 

__Performance considerations in distributed system__
Two kinds: 
* replica set
* sharded set

A Sharding cluster (is a horizontal scaling) consists of: 
* shard nodes: hold the application data. docs, indexs resides in shard nodes. They are replica sets. 
* mongos: used to route our client application requests to designated nodes. 
* config servers: these nodes are responsible for holding the mapping of our shard cluster, where data sits, and general config of sharding cluster
* Note: the way that your different hosts that hold the different nodes of your cluster are connected can affect overall performance of your system. 
* **Colating a mongos within the same server as your application to minimize the number of network hoops required to access shard nodes can reduce latency.**

Latency/workflow: the client application will talk to Mongos. Mongos will establish configurations with config servers to fetch all the configuration of all your shards, and with the shard nodes to retrieve info requested by the application. 

Two types of reads in sharding cluster: 
1. scatted gathered: ping all nodes for the data that your applications are requesting. will not use shard key, as the system will not be able to determine with exact precision which of the nodes contains the information that you need to satisfy your client's request
2. routed queries: ask one single shard node for all the data that your applications are requesting. use shard key. With the shard key, Mongos can pinpoint which shard nodes contains the information that you need to satisfy your client's request

Sorting/Skip and Limit in sharding cluster: sort/skip/limit will be performed within each node, and merge sort/skip/limit will occur in the primary shard of our data, finally return information to clients. 

Working with distributed systems: 
* consider latency
* data is spread across different nodes
* read and write implications (reads comes in in a different pace)

__Increate writing performance with sharding part__
* vertical scaling: scale your machine vertically by increasing RAM, I/O, CPU
* horizontal scaling: by splitting your work on several machines. 

Shard key: e.g., `sh.shardCollection('m201.people', {last_name: 1})`: shard key `{last_name: 1}` defines how our data is partitioned across different machines, it is either a index field, or index compound field (e.g., `{last_name: 1, state: 1}`) that **exists in every document in the collection**. With  shard key, our data is splitted into chunks, each chunk has inclusive lower bound, and exclusive upper bound. A chunk's max size must not exceed 64 MB. If exceeds, MongoDB will split it into smaller chunks. But if a chunk has the same lower and upper bound, we cannot split. There are many chunks exisiting in each of the shard in a cluster. 
    * 
        * cardinality: means no. of distinct values for a given shard key. High cardinality is good. Cardinality deterimines the number of chunks that exist in our cluster. 
        * frequency: even distribution for each value. 
        * rate of change: avoid using monotonically increasing/decresing values as shard keys. Because as field value grows, as it changes, it will go to the same shard.  
__Aggregation pipeline on a sharded cluster__
* how it works?  Generally merging will happen on a random shard, but there are several exceptions. When **use $out, $facet, $lookup, $graphLookup, the primary shard will do the merging results**. Because if we're running these operations very frequently, then the primary shard will be under a lot of loads than other shards, degrating the benefits of our horizontal scaling. 
* where operations are completed. 
* optimizations

__Replica set: read from secondaries__
* can define read preference. by default, this is on the primary node. But you can re-define to change to other nodes. e.g., `db.people.find().readPref("secondary"), db.people.find().readPref("secondaryPreferred")`
* by default, read and write will be on the primary node. If you change read to "secondary", your read will be routed to one of the secondaries. Write, however, can only be routed to the primary node. If change read to "secondaryPreferred", will read from one of secondaries unless there aren't any available, in which case your reads will be routed to the primary. If change read to "nearest", it will read from the member which has the lowest network latency. The driver measures latency by  measuring network lag from the heartbeat message. Since write always goes into "primary", if read from "primary", gurantees you get latest version of data, this behavior is called strong consistency. However, if read from "secondary", no such gurantee, because data on the primary are replicated to secondaries asynchronally, this is called eventual consistency. 
* when reading from secondary is a good idea? 
    * when doing analytics/reporting. e.g., product app is still reading from primary, but analysts are doing analytics in the background on the secondary. Unpredictable queries can often perform collection scans, so using a secondary to run these queries will protect the Primary from ejecting the working set while doing collection scans.
    * provides reads with lower latency. If your application needs the reads to happen very quickly and you have customers in many regions of the World, performing reads on the nearest secondary may help attain this goal. Just note that the reads may be faster, however, they are more likely to get staled compared to reads performed against the Primary.
* when is a bad idea? 
    * "To increase performance in a write-heavy system". This is a false notion that when writes are overloading primary, can offload reads to secondary nodes. That is not true as writes comes to primary will be replicated to secondaries. So all members in the replica set have roughly the same amount of write traffic. 

__Replica sets with differing indexes__
* create a index on a secondary. this index is only used if we run a query on the designated secondary node. So primary node will show "COLLSCAN". 
* all nodes should be configured homogeneously. However, it is great for a few use cases like analytics/reporting on secondary nodes or text search to having a specific index on the secondaries alone.  
* caveats: should prevent the secondaries that hold those indexes to ever become primaries. If they do, given the nature of the indexes, the expectedperformance of your system, especially in what right workload is concerned, will be significantly impacted. Because if your primary node steps down, then your main appliation could then begin communicating with secondary members whose indexes are not designed to serve these queries. 

* A secondary should never be allowed to become primary
    - True! If we were to allow it to become primary our application will experience the different set of indexes, once it becomes primary. That will potentially affect your application's expected performance.
* These indexes can only be set on secondary nodes
    - False! The indexes can be set on the primary node, however we avoid doing so to prevent any impact on the operational workload, since these only service an analytical workload.
* We can create specific indexes on secondaries, even if they are not running in standalone mode
    - False! No we first need to safely shutdown the secondary, and then restart it in standalone mode before we can create an index on it.


__Final Exam__
1. By default, the explain() command will execute your query. No. 
3. If no indexes can be used then a collection scan will be necessary. Yes. 

5. Query plans are removed from the plan cache on index creation, destruction, or server restart. Yes. 

7. Running performance tests from the mongo shell is an acceptable way to benchmark your database. No, use open-sourced public available sources. 
8. Indexes can only be traversed forward. No. 
9. The ideal ratio between nReturned and totalKeysExamined is 1. Yes. 
10. You can use the --wiredTigerDirectoryForIndexes option to place your indexes on a different disk than your data. Yes. 
11. Creating an ascending index on a monotonically increasing value creates index keys on the right-hand side of the index tree. Yes. 
12. Write concern has no impact on write latency. No. Since if write to secondaries, it needs longer time to return results to clients. 
13. You can index multiple array fields in a single document with a single compound index. No. Array fields create one index for each item in the array. 
14. Covered queries can sometimes still require some of your documents to be examined. No. covered queries no need any document examined, one of benefits of a covered query. 
15. A collection scan has a logarithmic search time. No. 
16. Under heavy write load you should scale your read throughput by reading from secondaries. No. 
17. Indexes are fast to search because they're ordered such that you can find target values with few comparisons. Yes. 
18. On a sharded cluster, aggregation queries using $lookup will require a merge stage on a random shard. No. 
19. When you index on a field that is an array it creates a partial index. No. 
20. Indexes can solve the problem of slow queries. Yes. 
21. MongoDB only allows collations to be defined at collection level. No. 
22. Collations allow the creation of case insensitive indexes. Yes. 
23. Creating an index with a different collation from the base collection implies overriding the base collection collation. No. it doesn't erase the base collecton collation. 
24. We can define specific collations in an index. Yes. 
25. It's common practice to co-locate your mongos on the same machine as your application to reduce latency. Yes. 
26. By default, all MongoDB user-created collections have an _id index. Yes. 
27. Collations can be used to create case insensitive indexes. Yes. 
28. Background index builds block all reads and writes to the database that holds the collection being indexed. No. MongoDB uses an optimized build process that only holds the exclusive lock at the beginning and end of the index build. The rest of the build process yields to interleaving read and write operations.
29. MongoDB indexes are markov trees. No, b-tree. 
30. Indexes can be walked backwards by inverting their keys in a sort predicate. Yes. 
31. It's important to ensure that secondaries with indexes that differ from the primary not be eligible to become primary. Yes. 
32. Indexes can decrease insert throughput. Yes. Each index on a collection adds some amount of overhead to the performance of write operations. For each "insert" or "delete" write operation on a collection, MongoDB either inserts or removes the corresponding document keys from each index in the target collection.
33. It's important to ensure that your shard key has high cardinality. Yes. 
34. Partial indexes can be used to reduce the size requirements of the indexes. Yes. 

__Data Modeling__
* it is a way to organize fields in a doc to support your application performance and querying capabilities. 
* main concern: 1. what data to store 2. how it will be quried. 
__Install mongodb on Windows__
1. create the data folders to store your databases
2. setup alias shorts for mongo and mongod. 
    - open GitBash
    - Change dir to your home dir by `cd ~`
    - create bash_profile `touch .bach_profile`
    - put in the following in .bash_profile
    - 
    ```
    alias mongod="C:/Program\ Files/MongoDB/Server/4.2/bin/mongod.exe"
    alias mongo="C:/Program\ Files/MongoDB/Server/4.2/bin/mongo.exe"
    ```
    - verify the setup by 
        - close down current terminal
        - re-launch git bash
        - type `mongo --version`, you shall see mongoDB shell version, build env, etc


__Install MongoDB on Mac__
* Follow the [steps](https://docs.mongodb.com/manual/tutorial/install-mongodb-enterprise-on-os-x/), at step 3 (Set permissions for the data and log directories.), change "my_mongodb_user" to be the username of your macbook, e.g., casiebao. 


## MongoDB Aggregation 
* `db.<collection>.aggregate([{stage 1}, {stage 2}, ...], {options})`
* aggregation operators vs query operators
* Field Path: "$fieldName", System Variable "$$UPPERCASE" (e.g.,$$ROOT	References the root document), User Variable "$$lowercase", some expr let us temporarily bind a value to a name or provide us a special name to access some data. 
* **All except the $out, $merge, and $geoNear $indexStats stages can appear multiple times in a pipeline.**

__$match__
* a match stage may contain a $text query operator, but it must be the first stage in a pipeline
* $match should come early in an aggregation pipeline
* you cannot use $where with $match
* $match uses the same query syntax as find

e.g., 

__$project__
* project can 1. retain fields: select the fields to display, 2. rename fields `db.solarSystem.aggregate([{$project: {_id: 0, name: 1, surfaceGraviry: "$gravity.value"}}])` 3. assign new fields. $project can be used as many times as in the aggregation pipeline. 
* retain fields: 
    * use only 1s: e.g., `.find({query}, {<field1>: 1, <field 2>: 1})`: only fields specified plus "_id" field will be popped out
    * use only 0s: e.g., `.find({query}, {<field1>: 0, <field 2>: 0})`: fields didn't get specified show up. 
    * exception: e.g., .`find({query}, {<field1>: 1, "_id": 0})`: combined 0s and 1s only only if "_id" is included in projection. 
    * except using 0s and 1s, aggregation also can `db.solarSystem.aggregate([{"$project": {"myWeight": "$myWeight"}}])`
* `$divide: ["$gravity.value", 9.8]`
* `$multiply: [{$divide: ["$gravity.value", 9.8]}, 86]`
* `db.solarSystem.aggregate([{$project: {myWeight: {$multiply: [{$divide: ["$gravity.value", 9.8]}, 86]}}}]`
* $size, $split

__Utility Stages__
* addfields
* setIntersection: intersection of two or more arraies. 
* sample: if N<=5% of num of docs in collection AND collection has >= 100 docs AND $sample is the first stage, then peudo-random cursor to randomly select N docs. Elsewise, in-memory random sort, select N docs, and this sort is subject to the same memory cost as in the sort stage. 
* size: size of an array

__geoNear__
* deal with geo JSON data
* Note that some fields in geoNear are required. 
* The collection can have one and only one 2dsphere index
* If using 2dsphere, the distance is returned in meters, if using legacy coordinates, the distance is returned in radians.
* $geoNear must be the first stage in an aggregation pipeline. 

__Cursor-like stages__
* `db.collection.aggregate([{$project}, {$limit: 5}], {allowDiskUse: true})`
* limit(), sort(), skip(), count()
* if sort not based on index field, it performs a in-memory sort, which are limited to 100M of RAM by default. If handling large dataset, need to add `allowDiskUse: true`. 

__$group__
* $sum
* `$cond: [{<if condition>}, {<then expr>}, {<else expr>}}]`: e.g., `$cond: [{$isArray: "$directors"}, {$size: "$directors"}, 0]`: note here need to use cond on size cuz it throws an error when array field is not existed in a doc. 
* [Accumulator operator](https://docs.mongodb.com/manual/reference/operator/aggregation/group/) is one of the following operators: e.g., avg, first, last, max, min, push, sum, stdDevSamp etc. Within $project, these operators will not carry their values forward and operator across multiple documents, thus have no memory between docs. 
* Accumulator expressions ignore doc with a value at the specified field that isn't of the type or if the value is missing. 
* group can be used multiple times in a pipeline
* $addToSet

*`{"$group": {"_id": "$name", "count": { "$sum": 1 }}}`: count number of items in this group

__Unwind__
* $unwind only works on array values
* there are two forms unwind, short form and long form
* using unwind on large collection may raise memory issues, so always match first to reduce size. 

__$lookup__
* "from" collection cannot be sharded
* "from" collection must be in the same db as the current collection

__$graphLookup__
* allows to look up recursively a set of documents with a defined relationship to a starting doc
* $maxDepth takes only Long values (Long values mean 64-bit integers)
* Say maxDepth=N, in self-lookup, that is (N+1) levels, in cross-collection lookup, that is N levels. 
* memory allocation: graphLookUp may need to allocate a lot of memories due to its recursive nature and the complexity of lookup, so could use $allowDiskUse. Even if you use $allowDiskUse, it may also exceed the 100MB maximum allowed for the aggregation pipeline. 
* Indexes: having "connectToField" being in indexes is a good practice. 
* Sharding: cannot have sharding in our "from" collection. 
* $graphLookup can be used in any position of the pipeline
* $match will not save the resources if the matched fields are unrelated to the graphLookup. 

__Facets: Multi-dimensional Grouping__
* Facet: allow users to explore data by applying multiple filters
* nested facets is not allowed
* A single query facet: it does group any particular data dimension. e.g., `db.companies.aggregate([{'$match': {'text': {'$search': 'network'}}}, {'$sortByCount': '$category_code'}])`: 
$sortByCount groups incoming docs coming from our match query, and then compute the counts of documents in which distinguish group, each group is a doc with two fields, `{"_id": group name, "count": count for this group}`, and display outputs by their sorted count.
* MongoDB allows multiple $facet stages in a pipeline  
* Manual buckets: 
    * Bucket and group by number of employees. Results have two fields, `"_id": <bucket>, "count": <number of employees in this bucket>`. If the specified field is not a numerical type, or fall outside of these buckets, then an error arises. 
    ```
        db.companies.aggregate([
            {'$match': {'founded_year': {'$gt': 1980}, 'number_of_employees': {'$ne': null}}},
            {'$bucket': {'groupBy': '$number_of_employees', 'boundaries': [0,20,50, 100, 500, 10000, Infinity]}}
        ])
    ```
    * add a "Other" field to group documents that don't have the specified field or it falls outside of the described boundaries.
    ```
    db.companies.aggregate([
        {'$match': {'founded_year': {'$gt': 1980}},
        {'$bucket': {'groupBy': '$number_of_employees', 'boundaries': [0,20,50, 100, 500, 10000, Infinity], 'default': 'Other'}}
    ])
    ```
    * Must always specify at least 2 values to "boundaries"
    * all items in the "boundaries" array must be in the same data type, boundaries items could be numerical, or strings such as ["a", "asef", "z"] as long as they have the same type. 
    * "count" is inserted by default with no "output", but removed when "output" is specified. Outputs have two fields `"{_id": <bucket value>, "count": },` plus fields defined in the ouput option. For manual buckets, `<bucket value>` is the boundary value for this bucket; for automatic buckets, `<bucket value>` is `{min:, max: }` auto-generated by the system. 
    * Automatic Buckets: 
        * $bucketAuto accepts an optional $granularity which ensures the boundaries of all buckets adhere to a specified preferred number series. 
        ```
        db.companies.aggregate([
            {'$match': {'founded_year': {'$gt': 1980}},
            {'$bucketAuto': {'groupBy': '$number_of_employees', 'buckets': 5， 'granularity': 'R20', 'output': {'total': , 'average': }}}
        ])
        ```
* Multiple Facets: The $facet stage allows several sub-pipelines to be executed to produce multiple facets. Each sub-pipeline within facet is past the exact same set of input docs that the match stage generated, and they are indep of one and another. The output for one sub-pipeline cannot be used for other sub-pipelines within facet. 

__$redact__
* $redact stage: useful to implement access control. But is not for restricint access to a collection
    * $$DESCEND: retain all fields at the current document level being evaluated, except for sub-documents and arrays of documents. It will instead traverse down, evaluating each level
    * $$PRUNE: remove all fields in the current document level w/o further inspection
    * $$KEEP: retain all fields in the current document level w/o further inspection
* If comparing a field in a doc, the field must be present at every level using $$Descend or deciding what to do if the field is missing, otherwise, $redat throws an error. 

__$out__
* will create a new collection or overwrite an exisitng unsharded collection if specified
* create collections in the same db as the source collection
* if a pipeline with $out errors, it will not write/create a new collection 
* it must be the last stage of the aggregation pipeline, thus cannot use within $facet to generate many differently shaped collections. 
* it must hornor exisitint indexes if specifying an existing collection. 

__$merge__
* last stage in the pipeline, giving yours options to specify whether whether you want to merge or insert into a new or exisiting collections in the same or different database. 
* can merge into a new or existing collection in the same or different db
    * `{$merge: {db: "db2", "coll": <collection name>}}`: a diff db
    * `{$merge: <collection name>}`: current db
* merge on 
    * specify a "on" field `{$merge:{into: <target>, on: [<field 1>, <field 2>]}`
    * default: `on: "_id"` for unsharded collection, `on: _id and shard key` for sharded collection.
* Optional params: 
    * `{$merge: {into: <collection name>, let: {itotal: "$total"}, whenNotWatched: , whenMatched: }}`
    * whenNotWatched options: insert, discard, fail
    * whenMatched options: merge, replace, keepExisiting, fail, [...].
    * [...]: you can custom your option e.g., whenWatched, $addFields of $total (fields in the existing doc), with $$new.total: $$new refer to incoming document
    * let option: how you want to call your incoming doc. here create a new field called itotal to represent the "total" field of the incoming document. Without let specified, by default, let: {new: "$$ROOT"}, that's why $$new refer to the incoming document. 

__$views__
* views are read-only.  
* views contain no data themselves (it contain no document), they are stored aggregations that run when queried to reflect the data in the source collection (i.e., the documents "in" a view are simply the result of the defining pipeline being executed). 
* views have some restrictions, and they cannot contain find() projection operators
* horizontal slicing is performed with $match stage to select the number of documents
* vertical slicing is performed with $project to reduce the number of fields returned. 
* view performance can be increased by creating the proper indexes on the source collection. 
* Create the view using command createView: `db.createView(<view>, <source>, <pipeline>, <collation>)`


__$indexStats__
* To view statistics on the index use on the orders collection, run the following aggregation operation:
`db.orders.aggregate( [ { $indexStats: { } } ] )`, the operation returns stats regarding the use of each index for the collection, it contains info such as key (index key specification), shard (name of the shard associated with the host)
* it must be the first stage in an aggregation pipeline, and cannot be used in "$facet" stage. 

## Aggregation Performance 
__Realtime processing vs batch processing__
* realtime processing: provide data for applications, query performance is more important
* batch processing: use aggregation for analytics, query performance is less important

Aggregation optimization: 
* use index as often as possible
* index are used in the pipeline from top to bottom until a stage where it cannot use index, and the following stages will stop using index (e.g., transforming data in a pipeline stage prevents us from using indexes in the stages that follow)
* put sort stage to front stages, put limit stage near sort stage
* when $limit and $sort are close together, a very performant top-k sort can be performed. 
* results are subject to 16MB document limit
* for each stage in pipeline, therer is 100 MB of RAM limit. To mitigate it, use indexes, or allowDiskUse:true (but using disk will seriously reduce performance, so it is used in batch processing than realtime processing)
* Explain output: `db.collection.aggregate([], {explain: true})`, from explain output, we don't want to see fetch stage under "winningPlan" because it means MongoDB needs to go to documents rather than using info from index alone. 
* try to avoid needless projects
* use accumulator expr, $map, $reduce, $filter in project stages if you want to group within a doc, rather than across documents, before an unwind
* every high order array function can be implemented with $reduce if the provided expr do not meet your needs
* the aggregation framework will auto reorder stages in certain conditions
* causing a merge in a sharded deployment will cause all subsequent pipeline stages to be performed in the same location as the merge
* the aggregation framework can auto-project fields if the shape of the final document is only dependent upon those fields in the input document
* the query in a $match stage can be entirely covered by an index. 

__Aggregation pipeline on a Sharded cluster__
* data are across different shards, so mongodb server determine which stage need to be executed in each shard, and what stages need to be executed on a single shard where the results from the other shards will be merged together. Generally, the merge will happen on a random shard. Except for $out, $faucet, $lookup, $graphLookup, for these queries, the primary shard will do the merging. If you use these operations very frequent, the primary shard requires more work loads than other shards, which reduces horizontal scaling. To mitigate, that, use a machine with more resources for your primary shard. 

## MongoDB Performance
__Hardware Consideration and Configuration__
* Mongo uses RAM for 1. aggregation 2. index traversing 3. write operations 4. query engine 5. connections
* MongoDB mainly uses CPU for 1. storage engine 2. concurrency model (A concurrency model specifies how threads in the the system collaborate to complete the tasks they are are given.)

## Basic Cluster Administration
## The Mongod
__Configuration file__
* Configuration file in written in YMAL
* How to set up configuration in command line
    * point to your own mongod file `mongod --config "/etc/mongod.conf"`
    * modify the default conf `mongod -f "/etc/mongod.conf"`

__File Structure for /var/lib/mongodb/__
* .wt files: collection data and index data. Even with new mongodb db, typically there are a few dbs and collections by default, so you shall always see some .wt files. 
    * collection-x-xxxxxx.wt: collectiond ata
    * index-x-xxxxxx.wt: index data


__Basic Commands__
* `db.serverStatus()`: database status
* `db.runCommand({<command>})`: mongodb shell to run a database command

__Logging Basics__
* Log verbosity levels: 
    * -1: inherit from parent
    * 0: default verbosity to include informational messages
    * 1-5: increases the verbosity level to include debug messages
* access the logs: 
    * db.adminCommand({ "getLog": "global" }) from the Mongo shell
    * tail -f <path-to-log-file> from the command line
* Log message Severity levels: Fatal, Error, Warning, Informational, Debug

__Profiling the database__
* when profiling enables, it restore data for all operations on a given database into a new collection called "db.system.profile"
* Events captured by the profiler: 
    * CRUD
    * Administrative operations
    * config operations

__Basic Security__
* You should always deploy MongoDB with security enabled, regardless of the environment.(development, evaluation, deployment etc. )
* MongoDB security is built on Authentication and Authorization
    * Client Authentication :
        * SCRAM: password security
        * X.509: uses X590 certificate for security, an option in MongoDB Extrepise. 
        * LDAP and KERBEROS: MongoDB Enterprise only
    * Cluster Authentication: 
        * a secret handshake between clusters
    * Authorization: role based access control. Each user has one or more Roles, and each Role has one or more Previleges. e.g., db admin can Create User and Create Index, and developer can Write/Read data, and data scientist can only Read data. 

## Python Client

## BSON
`pip install bson`: for BSON (binary JSON) encoding and decoding

### Pymongo 
__Installation__
Note: need to install bson first, then install pymongo to avoid some errors

__Read__

* reading with `find_one()`: returns a record directly; reading with `find()`: return an iterable, need to iterating through cursors to read records. 
* use cursor to save results as Python iterables. Note that the cursor object expires after being called once. 
* we here use bson.json_util to dump cursor object and get it printed nicely
* you could also use `list(cursor)` to return a list of records contained in the cursor. 
* using "cast.actor" to project sub-field of "cast" field

__Cursor Methods and Aggregation equalvalents__
Aggregation: It is a MongoDB operation. Basically it is composed of a pipeline of operations (a list of stages of query operation dict) to query records from db. Pymongo also has aggregate methods by `<collection>.aggregate(<pipeline list>)`. 

Cursor methods are methods available in pymongo. The following cursor methods have MongoDB equals: 
* limit(n): number of outputs
* sort([("field name", "sorting order"), (...)])
* skip(n): skip the first n items. 

__Writes__
* insert_one: the return object contains "_id" of the inserted object, checked by `inserted_result.inserted_id`, and it tells whether the operation is acknowedged by the server, checked by `inserted_result.acknowledged` which returns a Boolean value. 
* upsert version in updates: Sometimes, we want to update a document, but we're not sure if it exists in the collection. We can use an "upsert (meaning update and insert)" to update a document if it exists, and insert it if it does not exist.
This operation may do one of two things:
    * If the predicate matches a document, update the document to contain the correct data.
    * If the document doesn't exist, create the desired document.
    ```
    upsert_result = vg.update_one( { "title": "Fortnite" } , { "$set": fortnite_doc }, upsert=True )
    upsert_result.raw_result
    ```
    * this result object should have 'updatedExisting': True, because this operation updated an existing document. If it is insertion of a new record, 'updatedExisiting': False. 
    * besides, ".raw_result" include information such as number of updates, operationTime, ObjectId, clusterTime etc. 
    * "update_one({query}, {update operation}, upsert=True)". In this example, the query predicate here is { "title": "Fortnite" }. 
    * if have multiple items in the query or update operation, put them all in the curl bracket, e.g., use the user_email and comment_id to select the proper comment, then
    update the "text" and "date" of the selected comment. 
    `response = db.comments.update_one(
        { "email": user_email, "_id": ObjectId(comment_id)},
        { "$set": { "text": text , "date": date}}
    )`
* updates: update operation (update_one, update_many) return a UpdateResult. acknowledged, matched_count, modified_count, and upserted_id. modified_count and matched_count will be 0 in the case of an upsert.  

__Write concerns__
{w: n}: where n is the number of nodes get committed before the client receives acknowledgement. 
* {w: 1}: default writeConcern, it makes sure writes have been commited by at least 1 node. 
* {w: 'marjority'}: requests acknowledgement made from server to client if the majority of nodes in the replica set applied the write from the primary db. It takes longer than {w:1}, and it is more durable because it ensures vital writes are majority-committed. 
* {w: 0}: don't request an acknowledgement that any nodes applied the write (i.e., client could get acknowledgement before data is actually written to any db - primary or secondary dbs). It is fastest writeConcern level, and least durable. 

`{w:n, j:<bool>}`: the j option to request acknowledgment that the write operation has been written to the **on-disk** journal

Set write concern
* for client-wise: `client = MongoClient(w="majority"), then check client.write_concern -> WriteConcern(w=majority)`
* for db: `from pymongo import WriteConcern, db = client.get_database("my_database",write_concern=WriteConcern(w="majority"))`
* for collection `from pymongo import WriteConcern coll2 = collection.with_options(write_concern=WriteConcern(w="majority"))
oid = coll2.insert({"a": 2})`

__Bulk writes__

Main idea: you have a sequence of write operations, each depending on preceding ones, if send them one by one will cause a lot of latencies, thus bulk them together, send the entire batch to MongoDB and get only one acknowledge back for the whole batch. Make multiple writes more efficient. 

* Ordered Bulk Write: 
    * default in Mongo
    * it executes write sequentially, will end execution after first write failure. 
* Unordered Bulk Write: 
    * has to specify with the flag "{ordered: false}"
    * executes writes in parallel
    * single failure won't prevent other writes from proceeding 

Question:  Ordered bulk operations are faster than unordered. False statement. 

An example of ordered bulk write in pymongo: 

# here's where the bulk operation is sent to MongoDB
bulk_results = mflix.movies.bulk_write(bulk_updates) ``` * bulk_updates: a sequence/list of pymongo operations * use "bulk_write" to feed in bulk_updates. 

pipeline = [
            {
                "$match": {
                    "_id": ObjectId(id)
                }
            }
        ]
        pipeline.extend([
            {
                '$lookup': {
                    'from': 'comments', 
                    'let': {'id': '$_id'}, 
                    'pipeline': [
                        {
                            '$match': {'$expr': {'$eq': ['$movie_id', '$$id']}}
                        }, 
                        {
                            '$sort': {'date': -1}
                        }
                    ], 
                    'as': 'comments'
                }
            }, 
            ])

        movie = db.movies.aggregate(pipeline).next()

pipeline = [
    {
        '$group': {
            '_id': '$email', 
            'count': {'$sum': 1}
        }
    }, 
    {'$sort': {'count': -1}},
    {'$limit': 20}
    ]

try:
    with inventory.watch(full_document='updateLookup') as change_stream_cursor:
        for data_change in change_stream_cursor:
            print(data_change)
except pymongo.errors.PyMongoError:
    print('Change stream closed because of an error.')