COMMitMDE'18: Eclipse Hawk: model repository querying as a service
•
1 like•133 views
Eclipse Hawk provides scalable querying of models by indexing them into graph databases. It addresses challenges of collaborative modeling on large systems by distributed teams. The Hawk API is designed for flexibility, performance, and scalability through features like multiple communication styles, efficient encodings, and paged results.
Report
Share
COMMitMDE'18: Eclipse Hawk: model repository querying as a service
- 1. Eclipse Hawk: model repository querying as a service Antonio García Domínguez // @antoniogado October 15, 2018 School of Engineering and Applied Science, Aston University
- 2. Introduction
- 3. Who am I? Current roles • 2011: Eclipse Committer (Epsilon, now Hawk) • 2016: Lecturer at Aston University (Birmingham, UK) • 2018: Technical Director at Beautiful Canoe (part of Aston) Research trajectory • Started with automated theorem provers (ACL2, anyone?)... • ... moved to software testing (e.g. mutation testing) • ... applied it to service-oriented systems (SOA, WS-*) • ... ended up in MDE land! (went from applications then to infra) • Now interested in NoSQL technologies (graph DBs, indexing) 1
- 4. Two of the challenges in collaborative MDE Larger, more distributed teams • Models are created by multi-disciplinary teams • Teams can be distributed across the world • Different members work on different parts of the model More detailed models of larger systems • We are modelling increasingly larger systems: • Entire buildings: BIM • Cars: AUTOSAR • We are increasing the level of detail of our models: • Executable models are on the rise • Statement-level reverse-engineering models • If not careful, tools become slow and inefficient! 2
- 5. Two existing collaborative approaches Eclipse Connected Objects (CDO) + Database storage on mature RDBMS (e.g. H2) + Per-element on-demand loading, can reuse SQL query planners + Referential integrity is well understood in RDBMS − Best performance requires careful mapping (e.g. ELists) − Hard to set up for first timers, requires specialist knowledge File-based version control (Git/SVN) + Industry already trusts Git/SVN with their code + Can use close-fitting data model, less manual mapping involved + Inspecting, backing up and managing files is well understood / Need to fragment large models into multiple files − Preserving inter-fragment links can be difficult − Running a query may require large download + loading all files! 3
- 6. Solving some of the issues with the file-based approaches Building the fragmentation into the editor • This could be one more service of the editor generator • EMF-Splitter (A. Garmendia et al.) does this for EMF Preserving inter-fragment links • Editor should reconcile refs when fragments are moved • Editor should warn about broken refs after deletions • Epsilon Concordance (Kolovos et al.) does this for EMF 4
- 7. Solving some of the issues with the file-based approaches Building the fragmentation into the editor • This could be one more service of the editor generator • EMF-Splitter (A. Garmendia et al.) does this for EMF Preserving inter-fragment links • Editor should reconcile refs when fragments are moved • Editor should warn about broken refs after deletions • Epsilon Concordance (Kolovos et al.) does this for EMF Scalable querying as a service • We could mirror each fragment into graph DB, then reconnect • Industrial graph DBs already used for big data applications • We can provide a remote API to run queries over the network • Hawk does this for EMF/UML/Modelio... (this talk!) 4
- 8. Hawk basics
- 9. Indexing a library model with Hawk We go from these model files... 5
- 10. Indexing a library model with Hawk ... to these NoSQL graphs. 5
- 11. Indexing a library model with Hawk • Ecore packages → metamodel nodes • Ecore classes → type nodes 5
- 12. Indexing a library model with Hawk • Physical files → file nodes • Model elements → element nodes 5
- 13. Indexing a library model with Hawk • MM index: package URI → metamodel node • File index: file path → file node • Users can define custom indices by attribute (e.g. title) or expression (number of books, “is singleton?”) [1] 5
- 14. Component-based architecture Hawk CoreHawk Core Model Parsers Model Parsers Storage Backends Neo4J OrientDB Ecore XMI IFC2x3 Modelio Version Control Systems Version Control Systems Subversion Folder / Git IFC4 Eclipse Workspace BPMN Storage BackendsClients Eclipse UI Thrift API Eclipse .MF HTTP Locations Query Engines EOL OrientDB SQL Greycat Time-aware EOL 6
- 15. Ways to deploy Hawk As an Eclipse plug-in • Aimed at proficient modellers, easy to iterate over • Provides Eclipse-based GUI and EMF resource abstractions • Can be integrated with other tools (e.g. EMF-Splitter) As a plain Java library • For solution developers: Hawk is hidden from the user • Most of Hawk is Eclipse-independent As a remote API • Remote querying service over shared model repositories • Much faster than downloading + loading the models 7
- 17. Search within collaboration server (SOFTEAM) [3] Modelio screenshot: http://modelio.org • Constellation: collaboration server for Modelio models • Needed search, but Modelio is entirely file-based • Integrated Hawk into the Constellation WAR file: reported that initial indexing cost quickly pays off 8
- 18. Search within collaboration server (SOFTEAM) [3] Constellation dashboard • Constellation: collaboration server for Modelio models • Needed search, but Modelio is entirely file-based • Integrated Hawk into the Constellation WAR file: reported that initial indexing cost quickly pays off 8
- 19. Search within collaboration server (SOFTEAM) [3] Query box for Constellation • Constellation: collaboration server for Modelio models • Needed search, but Modelio is entirely file-based • Integrated Hawk into the Constellation WAR file: reported that initial indexing cost quickly pays off 8
- 20. Search within collaboration server (SOFTEAM) [3] 1 2 3 4 5 6 ·105 200 400 600 800 1,000 1,200 Project size (model elements) Indexingtime(s) 500 1,000 1,500 2,000 2,500 OrientDBdiskspaceusage(MB) Time Disk space Indexing times and index sizes (OrientDB backend) 0 1 2 3 4 5 6 7 ·105 100 101 102 103 Project size (model elements) Indexingtime(s) MT (run 1) MT (runs 2–5) HT (run 1) HT (runs 2–5) Code generation times: Modelio (MT), Hawk (HT) • Constellation: collaboration server for Modelio models • Needed search, but Modelio is entirely file-based • Integrated Hawk into the Constellation WAR file: reported that initial indexing cost quickly pays off 8
- 21. Software modernization (SOFT-MAINT) • Software is migrated from an old technology to a new one, keeping its functionality • SOFT-MAINT does a good part of the work as a model-to-model transformation • Large system models do not fit in the memory of a normal workstation Model Hawk XMI SLOC Ratio set1 4s 5s 1 413 0,8 set2 4s 5s 4 664 0,8 set3 9s 10s 53 704 0,9 set4 12s 71s 700 545 0,17 Execution times for the Java2SMM transformation 9
- 22. Building Information Models (UNINOVA): concept Uses for Building Information Models • UNINOVA is a Portuguese construction company. • Uses models to integrate everything they know about a building: floor plan, wiring, piping, doors/windows, and so on. • Models can be in the order of GB! 10
- 23. Building Information Modelling (UNINOVA): buying doors Small (100K) Medium (500K) Large (2.5M) BimQL (G) 1.50s 2.30s 37.80s Hawk (P) 0.13s 0.22s 0.16s Hawk (G) 1.50s 1.90s 6.80s Query times over building doors, depending on # of elements: P is only IDs, and G is IDs + geometry. • Hawk indexed BIM IFC models from the OpenBIM Server • Used Hawk to query their buildings and export model views • Model views = a specific floor, or just the electric wiring • Model views required implementing paged fetch in the API 11
- 24. Building Information Modelling (UNINOVA): area and export Query in Hawk to compute the area of a building 12
- 25. MEASURE software metrics platform [2] • ITEA3 MEASURE: European industry-academia consortium • https://itea3.org/project/measure.html • MEASURE created a platform which brings togethr metrics from various SDLC phases, providing analysis tools and dashboards • Hawk has been integrated as a metric provider for the models used in the anlaysis and design stage 13
- 26. Design of Hawk API
- 27. Wishlist for a good model querying API? (1/2) Performance • Should be efficient in size (good encoding + compression) • Should be efficient in time (avoid CPU-intensive encoding) • Should avoid overheads from complex protocols • Should reduce total roundtrip time (# of interactions) Flexibility • Should work on most languages (Java, C#, Python, JavaScript...) • Should allow for various communication styles: • Fast, few results: request-response • Slow, few results: submit + fetch, with optional cancelling • Fast, many results: request-response, then paged fetch • Slow, many results: submit + fetch, then paged fetch • Streaming: server-to-client messages (push notifications) • Should be friendly with firewalls 14
- 28. Wishlist for a good model querying API? (2/2) Security • Should implement authentication (API key or user/pass) • Should implement authorization • Authorization granularity should be as fine as possible • Actual security mechanism should be configurable Scalability Should be able to handle a large number of users without failing (bad) or producing a wrong number of results (much worse!) 15
- 29. Does the Hawk API meet all of them? No. But we meet quite a few! And some APIs fare worse... 15
- 30. Hawk API design namespace java org.hawk.service.api service Hawk { /* ... */ QueryResult query( 1: required string name, 2: required string query, 3: required string language, 4: required HawkQueryOptions options, ) throws ( 1: HawkInstanceNotFound err1 2: HawkInstanceNotRunning err2 3: UnknownQueryLanguage err3 4: InvalidQuery err4 5: FailedQuery err5 ) /* ... */ } General structure • Based on Apache Thrift: efficient, flexible RPC library • Given IDL file, generates client + server stubs • Hawk implements stubs and exposes them through: • TCP (multithreaded server) • HTTP (Jetty servlet) Flexibility • Thrift supports 4 encodings • JSON is the most flexible • Tuple is the most compact • Over 26 languages supported 16
- 31. But I don’t like writing IDL files... • Neither do we — that’s why we wrote Ecore2Thrift • Transforms annotated Ecore metamodels to Thrift IDL files • https://github.com/bluezio/ecore2thrift 17
- 32. Relative performance of Thrift encodings (1/2) • Hawk provides operations to fetch models over the network • Three modes: greedy sends entire model, lazy attributes starts with structure with no attribute values, lazy children sends roots • Thrift Tuple doesn’t do too badly against EMF binary, and it doesn’t require knowing the metamodel in advance Encoding No compression Greedy Lazy attr. Lazy children XMI 27884.85 – – EMF binary 11353.19 – – Thrift JSON 56946.78 12143.58 0.75 Thrift binary 32471.56 6801.65 0.60 Thrift compact 22082.97 4020.68 0.41 Thrift tuple 19763.98 3414.46 0.39 Initial fetch messages for GraBaTs’09 set1 model in KB. No compression. 18
- 33. Relative performance of Thrift encodings (2/2) • If we apply gzip compression, things change! • Even the JSON encoding is better than EMF binary Encoding Compressed with gzip Greedy Lazy attr. Lazy children XMI 1844.54 – – EMF binary 1683.98 – – Thrift JSON 1569.01 1007.34 0.40 Thrift binary 1474.33 975.79 0.39 Thrift compact 1329.32 924.25 0.34 Thrift tuple 1285.28 901.36 0.32 19
- 34. So far... Performance Should be efficient in size (good encoding + compression) Should be efficient in time (avoid CPU-intensive encoding) Should avoid overheads from complex protocols → TCP ? Should reduce total roundtrip time (# of interactions) Flexibility Should work on most languages (Java, C#, Python, JavaScript...) ? Should allow for various communication styles: ? Fast, few results: request-response ? Slow, few results: submit + fetch, with optional cancelling ? Fast, many results: request-response, then paged fetch ? Slow, many results: submit + fetch, then paged fetch ? Streaming: server-to-client messages (push notifications) Should be friendly with firewalls → HTTP 20
- 35. Impact of communication style Grace Hopper’s bundle of nanoseconds • You don’t want to hit the network more than necessary • Every message implies some latency • However, you don’t want huge messages either • Implement multiple styles: let the client choose! 21
- 36. Request-response style: nice and easy for small, fast queries namespace java org.hawk.service.api service Hawk { /* ... */ QueryReport timedQuery( 1: required string name, 2: required string query, 3: required string language, 4: required HawkQueryOptions options, ) throws ( 1: HawkInstanceNotFound err1 2: HawkInstanceNotRunning err2 3: UnknownQueryLanguage err3 4: InvalidQuery err4 5: FailedQuery err5 ) /* ... */ } Advantages • Easy to implement: one call • Easy to use: invoke and wait • No need to correlate msgs • Great for small, fast queries • Reliable in heavy concurrence Note QueryReport has server query time: allows for separating the network latency from the query time. 22
- 37. Request-response style: nice and easy for small, fast queries namespace java org.hawk.service.api service Hawk { /* ... */ QueryReport timedQuery( 1: required string name, 2: required string query, 3: required string language, 4: required HawkQueryOptions options, ) throws ( 1: HawkInstanceNotFound err1 2: HawkInstanceNotRunning err2 3: UnknownQueryLanguage err3 4: InvalidQuery err4 5: FailedQuery err5 ) /* ... */ } Disadvantages • Huge response will use up all the memory of the server • Client cannot cancel a slow query: no query ID to refer to Recommendation Set a max response size, bail out when we exceed that. 22
- 38. Submit + wait style: allows for remote query cancelling namespace java org.hawk.service.api service Hawk { string asyncQuery(...) void cancelAsyncQuery( 1: required string queryID ) QueryReport fetchAsyncQueryResults( 1: required string queryID ) } Advantages • Submit response is immediate • Client receives UUID of query • UUID is used to fetch results and cancel if taking too long Disadvantages • Until fetched, results will take up memory in the server • Has more RTT than plain request-response Recommendation There should be a maximum lifespan for the results, so memory is freed eventually. 23
- 39. Paged fetches for large loads namespace java org.hawk.service.api service Hawk { list<ModelElement> resolveProxies( 1: required string name, 2: required list<string> ids, 3: required HawkQueryOptions options, ) } Advantages • Can handle larger responses • Initial query: only element IDs • Next: fetch elements in batches Disadvantages • We spend RTT for each page • Eventually, message with element IDs may still be too large Options (both with tradeoffs) • Introduce stateful paging • Re-run query on paging 24
- 40. Streaming queries, server status: push notifications Updates on index state change • Users want to know what the index is doing from the UI • We do not want to poll the server unnecessarily • Our choice: Apache ActiveMQ (AMQP, WebStomp) • Users are given ActiveMQ queue details over the Thrift API Updates on model changes • Thrift API also allows watching indexed models over changes • Same approach, but queues can survive client disconnects Streaming queries: pending! • Some graph DBs (e.g. Orient, Neo4j) allow you to leave a query running and receive results as they appear • We would need to tweak our query engine for this... 25
- 41. Done with the first two! Performance Should be efficient in size (good encoding + compression) Should be efficient in time (avoid CPU-intensive encoding) Should avoid overheads from complex protocols → TCP Should reduce total roundtrip time (# of interactions) Flexibility Should work on most languages (Java, C#, Python, JavaScript...) / Should allow for various communication styles: Fast, few results: request-response Slow, few results: submit + fetch, with optional cancelling / Fast, many results: request-response, then paged fetch Slow, many results: submit + fetch, then paged fetch / Streaming: server-to-client messages (push notifications) Should be friendly with firewalls → HTTP 26
- 42. API security Access control over HTTP: Apache Shiro • Simple to integrate: 2 .jar files • Configurable through shiro.ini • Filters all incoming requests to servlets • We provide a security realm backed by a MapDB user database • Security is coarse: all users have same level at the moment Credentials storage: Equinox secure storage • Hawk needs to access password-protected VCS • Stored credentials should be adequately protected • In Eclipse-based environments, Hawk uses its secure storage • Passwords are encrypted in custom keyring, with random password (generated from /dev/random) 27
- 43. API scalability SoSyM paper on stress testing APIs [4] • Tried Hawk (Thrift), Mogwai (Thrift), CDO (Net4j) • Used both HTTP and TCP variants • Used 1 server and 2 client machines, emulated 1–64 clients HTTP vs TCP: not that bad unless you do something wrong... • CDO HTTP was 8x slower than CDO TCP! • CDO TCP uses bidirectional nature of TCP to stream results • CDO HTTP attempted to emulate this with polling: don’t do this! • Hawk/Mogwai HTTP was only 20% slower than raw TCP Concurrency can ruin your day • Under heavy thread contention, race conditions quickly appear • This may stress your networking library and backend... 28
- 44. Failed GraBaTs’09/TB queries during API stress-testing Query Tool Proto 1t 2t 4t 8t 16t 32t 64t OQ CDO HTTP 1 1 CS CDO/H2 HTTP 1 CS Hawk/O/EOL HTTP 1 CS Hawk/O/EPL HTTP 2 PL CDO/H2 HTTP 1 RS Hawk/O/EOL HTTP 1 RS CDO/H2 TCP 1 SN Hawk/O/EPL HTTP 1 SS Hawk/O/EPL HTTP 1 SS CDO/H2 TCP 1 SS Hawk/O/EPL TCP 2 1 29
- 45. Wrong GraBaTs’09/TB queries during API stress-testing Query Tool Proto 1t 2t 4t 8t 16t 32t 64t OQ CDO HTTP 3 8 22 17 1 CS CDO HTTP 2 2 1 2 3 4 CS CDO TCP 1 3 3 6 RS CDO HTTP 2 3 2 6 11 12 9 RS CDO TCP 6 3 3 10 15 28 32 SS CDO TCP 1 30
- 46. Reviewing the last two blocks of requirements Security Should implement authentication (API key or user/pass) Should implement authorization Authorization granularity should be as fine as possible Actual security mechanism should be configurable Scalability Hawk over Neo4j worked reliably in all configurations / Hawk over OrientDB failed some queries (may need to revisit) / Greycat backend has not been stress-tested yet... 31
- 47. Conclusion and roadmap Remote querying APIs are not simple! • Many MDE researchers forget to pay attention to this • Good remote API design rules still apply: • Be efficient, but also firewall-friendly • Be flexible in what you accept and strict in what you produce • Avoid denial of service (excessive CPU/RAM use) • Queries vary in nature: provide multiple communication styles! • Allow use from more than just your host language: • Having JavaScript will make it much easier to add a web UI • Python? C++? C#? They could all consume model query results... Things we want to add to Hawk Fine-grained security, paging over async queries, result streaming, horizontal scaling, smartly delegating work to clients...
- 48. Thank you! Eclipse Hawk: model repository querying as a service Antonio García Domínguez // @antoniogado October 13, 2018 School of Engineering and Applied Science, Aston University Two existing collaborative approaches Eclipse Connected Objects (CDO) + Database storage on mature RDBMS (e.g. H2) + Per-element on-demand loading, can reuse SQL query planners + Referential integrity is well understood in RDBMS − Best performance requires careful mapping (e.g. ELists) − Hard to set up for first timers, requires specialist knowledge File-based version control (Git/SVN) + Industry already trusts Git/SVN with their code + Can use close-fitting data model, less manual mapping involved + Inspecting, backing up and managing files is well understood / Need to fragment large models into multiple files − Preserving inter-fragment links can be difficult − Running a query may require large download + loading all files! 3 Solving some of the issues with the file-based approaches Building the fragmentation into the editor • This could be one more service of the editor generator • EMF-Splitter (A. Garmendia et al.) does this for EMF Preserving inter-fragment links • Editor should reconcile refs when fragments are moved • Editor should warn about broken refs after deletions • Epsilon Concordance (Kolovos et al.) does this for EMF Scalable querying as a service • We could mirror each fragment into graph DB, then reconnect • Industrial graph DBs already used for big data applications • We can provide a remote API to run queries over the network • Hawk does this for EMF/UML/Modelio... (this talk!) 4 Indexing a library model with Hawk ... to these NoSQL graphs. 5 Wishlist for a good model querying API? (1/2) Performance • Should be efficient in size (good encoding + compression) • Should be efficient in time (avoid CPU-intensive encoding) • Should avoid overheads from complex protocols • Should reduce total roundtrip time (# of interactions) Flexibility • Should work on most languages (Java, C#, Python, JavaScript...) • Should allow for various communication styles: • Fast, few results: request-response • Slow, few results: submit + fetch, with optional cancelling • Fast, many results: request-response, then paged fetch • Slow, many results: submit + fetch, then paged fetch • Streaming: server-to-client messages (push notifications) • Should be friendly with firewalls 8 Does the Hawk API meet all of them? No. But we meet quite a few! And some APIs fare worse... 9 Impact of communication style Grace Hopper’s bundle of nanoseconds • You don’t want to hit the network more than necessary • Every message implies some latency • However, you don’t want huge messages either • Implement multiple styles: let the client choose! 15 Failed GraBaTs’09/TB queries during API stress-testing Query Tool Proto 1t 2t 4t 8t 16t 32t 64t OQ CDO HTTP 1 1 CS CDO/H2 HTTP 1 CS Hawk/O/EOL HTTP 1 CS Hawk/O/EPL HTTP 2 PL CDO/H2 HTTP 1 RS Hawk/O/EOL HTTP 1 RS CDO/H2 TCP 1 SN Hawk/O/EPL HTTP 1 SS Hawk/O/EPL HTTP 1 SS CDO/H2 TCP 1 SS Hawk/O/EPL TCP 2 1 23 Conclusion and roadmap Remote querying APIs are not simple! • Many MDE researchers forget to pay attention to this • Good remote API design rules still apply: • Be efficient, but also firewall-friendly • Be flexible in what you accept and strict in what you produce • Avoid denial of service (excessive CPU/RAM use) • Queries vary in nature: provide multiple communication styles! • Allow use from more than just your host language: • Having JavaScript will make it much easier to add a web UI • Python? C++? C#? They could all consume model query results... Things we want to add to Hawk Fine-grained security, paging over async queries, result streaming, horizontal scaling, smartly delegating work to clients...
- 49. References i K. Barmpis, Seyyed Shah, and D. S. Kolovos. Towards incremental updates in large-scale model indexes. In Proceedings of the 11th European Conference on Modelling Foundations and Applications, July 2015. A. García-Domínguez, A. Abherve, K. Barmpis, O. Al-Wadeai, and A. Bagnato. Integration of Hawk for Model Metrics in the MEASURE Platform. In Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development, Funchal, Madeira, Portugal, January 2018. SciTePress.
- 50. References ii A. García Domínguez, K. Barmpis, D. S. Kolovos, M. A. A. da Silva, A. Abherve, and A. Bagnato. Integration of a graph-based model indexer in commercial modelling tools. In Proceedings of MoDELS’16, pages 340–350, Saint Malo, France, 2016. ACM Press. A. García Domínguez, K. Barmpis, D. S. Kolovos, Ran Wei, and Richard F. Paige. Stress-testing remote model querying APIs for relational and graph-based stores. Software Systems Modeling, pages 1–29, June 2017.