Migrate away from MongoEngine.

[amcgregor]

This isn't exclusively tied to Contentment, but Contentment is probably the most extensive use of MongoEngine by Marrow projects.

The following issues directly impact our use of the library, and since 0.10 there have been an absolutely astounding number of regressions and new bugs added which have had far-reaching effects. Here are the short lists.

• Increase Code Health MongoEngine/mongoengine#897 — Overall code health.
• Drop support for positional arguments when instantiating a document MongoEngine/mongoengine#2103 — Removal of contributed positional instantiation functionality.

Enhancements

• Promote and demote inherited classes. MongoEngine/mongoengine#167 (rejected) — A nearly four year old ticket directly relating to promotion and demotion of Contentment Asset classes to allow for post-creation specialization and migration between specializations.
• Make reference fields a proxy to allow access to field.id without dereferencing MongoEngine/mongoengine#298 — 2013. Allow access to referenced document ID without full dereferencing; similarly aged ticket.
• Cannot query on _cls MongoEngine/mongoengine#450 — Can't query on the generated class name field.
• Support more projection operators MongoEngine/mongoengine#529 — Projection positional operator. 2013. I mean, really.
• Document registry in mongoengine: class name as a unique Document identifier leads to failing unit tests. MongoEngine/mongoengine#1319 — Using model class name for unique identification is sub-optimal.
• Field order not preserved during redefinition. MongoEngine/mongoengine#1321 — Field order not preserved during redefinition.
• when declaring a non-default primary key, why cant I access it by the name of the field? MongoEngine/mongoengine#1293 — Can't define a primary key attribute name.
• MongoDB 3 Compression Options MongoEngine/mongoengine#1099 — Support MongoDB collection compression options in meta.

Performance

• Document instantiation performance MongoEngine/mongoengine#1137 (and Mongoengine is very slow on large documents compared to native pymongo usage MongoEngine/mongoengine#1230) — MongoEngine recursively converts documents on load, suffering performance issues on large documents due to this eager conversion.
• Efficiency of "in" with BaseList MongoEngine/mongoengine#1116 — List field in checks are quite slow.
• Major performance drop between 0.8.7 and 0.9.0 MongoEngine/mongoengine#1073 — Unaddressed drop from 0.8.7 to 0.9.0.
• QuerySet count vs len MongoEngine/mongoengine#937 — Queryset cache really kills things, especially len() performance.

Regressions

• Field where class field name != document field name is not being updated to database MongoEngine/mongoengine#1178 — Fields with overridden db_field (about 95% of ours) don't get marked as dirty and are not saved correctly.
• With context_managers.no_dereference, the query results still dereference data MongoEngine/mongoengine#1245 — Disabling dereferencing doesn't disable dereferencing.
• regression in 0.10.0 compared to 0.9.0 for queryset limit after count MongoEngine/mongoengine#1136 — An internally infamous bug, if you count the result of a query, limit is no longer applied.
• regression from 0.9.0: assert on reverse_delete_rule=CASCADE with source.id.field != dest.id.field MongoEngine/mongoengine#1135 — Cascade reverse delete rule issues.
• error while using "db_field" parameter in multi-level embedded documents MongoEngine/mongoengine#904 — A PR exists, but not accepted yet, to fix db_field usage in hierarchical embedded documents.
• save() may break database consistency (concurrency issue) MongoEngine/mongoengine#564 — Saving uses an upsert by default. This has certainly caused some of our harder to track down "where the hell did this mangled data come from?!" problems.

Bugs

• Cannot query and filter fields of subclasses of embedded documents MongoEngine/mongoengine#475 — Can't query/filter fields that are present on embedded document subclasses.
• Document.save not saving the doc MongoEngine/mongoengine#1246 — Use of primary_key is pretty broken.
• Reload doesn't check if attribute is writable  MongoEngine/mongoengine#1276 — Reload attempts to write attributes based on data, not current field lists.
• no_dereference context manager suppresses all exceptions on exit MongoEngine/mongoengine#1284 — Beyond not disabling dereferencing, the context manager also eats all exceptions. (As does the no_sub_classes manager.)
• switch_db does not dereference properly MongoEngine/mongoengine#1288 — Dereferencing in the context of multiple database connections fails.
• Please recall fix on: Saving document doesn't create new fields in existing collection #620 MongoEngine/mongoengine#1126 — Saving document doesn't create new fields in existing collection MongoEngine/mongoengine#620 incorrectly references fields by attribute name, not db_field.
• db_field isn't used if field is not present in document, leading to validation errors MongoEngine/mongoengine#1102 — Field not present, but defined with db_field, fails validation.
• Calling len() on a QuerySet object mid-iteration stops iteration at current hundred documents MongoEngine/mongoengine#1086 — Calling len() on a query set mid-iteration breaks iteration.
• If I use exclude I can lose my information MongoEngine/mongoengine#1085 — Critical data loss can occur if using projection.
• not__match__nin is not working MongoEngine/mongoengine#1034 — not__match__nin fails.
• mongoengine.errors.ValidationError: A ReferenceField only accepts DBRef or documents MongoEngine/mongoengine#997 — Can't safely reference abstract document base class.
• Connect blindly overrides username/password when URI style connection is used. MongoEngine/mongoengine#909 — URI overrides username/password on connect, even if it does not define a username or password.
• Method resolution order (MRO) error when Document is involved in diamond inheritance. MongoEngine/mongoengine#808 — Can't handle diamond inheritance. (This is huge for Contentment.)

Field Bugs

• EmailField doesn't support Unicode MongoEngine/mongoengine#1228 — Poor regular expression-based e-mail validation.
• decrementing Decimal raises InvalidDocument exception type MongoEngine/mongoengine#1320 — Atomic operations on decimal fields.
• QuerySetNoCache iterator protocol violation MongoEngine/mongoengine#1316 — Disabling of document cache breaks iteration protocol.
• Deleting an element from middle of ListField results in an incorrect list. MongoEngine/mongoengine#1318 — List fields fail to correctly track changes.
• fileField saved in default db even after switch_db() MongoEngine/mongoengine#1301 — GridFS file storage stores against default DB in multi-DB context.
• Causeless DecimalField behavior MongoEngine/mongoengine#637 — More decimal field issues.
• Wrong ObjectIdField to_mongo() and validate() methods behavior on Python 3 MongoEngine/mongoengine#618 — ObjectId fields behave badly on Python 3.
• Bug: ListField may lost value if do iterate before some edit operation MongoEngine/mongoengine#632 — List field data loss.
• ListField minus index assignment does not work MongoEngine/mongoengine#1119 — Can't utilize negative indices on list fields for assignment/update.
• Setter does not behave as it should. MongoEngine/mongoengine#1059 — More decimal field setter issues; transform to float.
• ImageField is nullified after save MongoEngine/mongoengine#847 — Image field nukes itself after retrieval and save.

Don't get me started on the potential and the failure of CachedReferenceField.

• AttributeError: 'CachedReferenceField' object has no attribute '_get_db' MongoEngine/mongoengine#1075 — Missing _get_db.
• CachedReferenceField creates DBRefs on to_python, but can't save them on to_mongo MongoEngine/mongoengine#1047 — Creates DBRef, can't save.

I can't even find back the ticket where I requested access to cached data without triggering dereferencing. (Was closed as "wont fix" long ago.) The vast majority of the above are directly impactful and/or have actually been encountered at Illico, forcing us to pin to < 0.10. Given that our database servers will be updated to 3.2 in the near future, with new getMore semantics, we can't stay pinned forever.

So yeah, marrow.schema and marrow.mongo are somewhat more important things moving forward.

[amcgregor]

Marrow Mongo is a thing now. Time to get the hands dirty with some refactoring…