Solar, a web experiment

As someone interested by tooling, I often think about how to reduce frictions between the developed applications and the development tools. In other words, how to reduce the context switching cost. For two weeks, I have started to work on a new experiment, named Solar, a play module to work on your play applications directly in your browser.

I have been largely inspired by Orion and Erwan module, but I had specific goals, that's why I did not reuse most of those projects code. Actually, I would like to focus on several concerns:

• simplicity with neither code completion nor refactoring
• quality with continuous testing and deployment integration 
• collaboration with code review integration 

Currently, I have implemented the following features:

• Navigation in the application files, with the ability to edit files and create new ones. 

• Java files edition with syntax highlighting and live compilation during the edition.

• Router configuration file edition with syntax highlighting. 

The editor is implemented in the canvas, and it is based on Guillaume's work.

Joining Zenexity

Last week, I have joined Zenexity. Joining a new company, is a great adventure. By meeting new people and working on different projects, you learn new things and discover other ways of doing things. This leads to impassioned discussions and knowledge sharing. On my side, I am starting to dig deeper into Scala.

Zenexity is the company who created the Play framework, a java web framework, which as you probably know, drops the craziness complexity of JEE to focus on simplicity. 

In a recent presentation, I have spoken about collaboration. I notably mentioned that in the last five years applications and medias which integrate natively collaboration have known impressive success. Looking at collaboration is interesting, as it exhibits today web application concerns.

Integration of multiple data sources

Enabling collaboration means dealing with several data producers. Typically, for each you will have a connection, and one of the application responsibility will be to manage those connections. The management becomes harder when the number of producer raises. For instance, the application may reach a connections number limit, so in order to scale the application will need to detect idle producers for closing temporally their connection.

Dealing with large amout of data

When you add more potential data producer, this generally ends with more information to manage or at least more information you would like to store, for instance for statistics purpose. To achieve scalability you may choose a non relational database.     

Realtime interactions

To ease collaboration, you need to provide a way for users to interact between them. Basically each user should be able to send notifications to others, directly or indirectly, and to do it efficiently traditional polling is not efficient.

The new coming release of Play tends to tackle those concerns. It will provide powerful abstractions which will enables you to write simply reactive web applications. Applications which, integrate multiple data sources, support multiple data stores, and could use efficient bidirectional communications.

Collaborative modeling @ EclipseCon Europe 2011

Last week I presented with Martin and Benjamin why collaboration in modeling becomes an important concern, the issues we currently encounter, and the eclipse technologies which are improving to provide seamless collaboration.

	In the last five years models usage generalizes, modeling tools matured a lot, but often they have not be designed to allow one to work efficiently in team. On the same time, collaboration, in a general manner, took a predominant place in tools and medias. Successes of Wikipedia and the Huffington Post show that trend.
	Modeling or not, working with others is a challenge. Which separation of work ? Which rules to set up to keep the coherence ? How to deal with the different concerns people may have ? Answers are often most organizational than technical, however tools, technologies may constraint your organization and you way to work in collaboration.
	Model together is first dealing with a large amount of information and depending of your manner to represent them it could become quite tricky. For graphical representations, beyond layout problems, from a certain number of elements, we are not able to focus on the useful information, the signal noise ratio becomes to low.
	Even without representation scalability remains an issue, as by default EMF load the entire model in memory.
	How many times are we not able to understand legacy code or models done by others ? It is necessary to document your models and explains your choice behind design decision, but it is as important to keep documentation synchronized across the changes and refactorings you made.
	To collaborate means dealing with concurrent accesses, and often end users do not want to have to compare or to merge. Starting from that requirement you quite easily ends up with some kind of pessimistic strategy.
	The most simple implementation of this strategy is an instant messaging discussion. As long as your model is not changing too often its bearable.  Sophisticated versions of that implementation are directly integrated in an SCM supporting file locking, but that remains a not very elegant solution.
	From that you'll try to avoid blocking the whole team and then you'll split your model into many files. It's slightly better but it's not that easy to do, as you have to carefully design your Ecore model avoiding most of the cross references. More over EMF fragments are not correctly supported by many tools.
	The other strategy, called the optimistic one, is to allow conflict happens and to deal with them when they occur. It looks like a dice game, and depending on the frequency, it could become hard to deal with conflicts.
	Those problems could be tackled with several technologies which enable one to work more naturally in collaboration.
	How to deal with large diagrams ? Thanks to EMF Mylyn bridge we are able to focus in the diagram on the information which matters, the information contextual to a change you have to do or that somebody else did.
	How keeping documentation and model synchronized ? Thanks to Mylyn Intent we could mix natural and formal language. This is some kind of literate programming adapted to modeling with one specificity : you can update the model, or the doc, it doesn't matter, the tool help you keeping them synchronized anyway
	How to support better models fragmentation ? Model tools could be behaving like the tools we are used to when programming. They should not  make the assumption that every referenced element will always be there. This is possible by playing nicely with the EMF proxy mechanism.
	Keeping models coherent with fragmentation could be ensured through platform logical models. That API enables components to trigger changes on any file operation.
	How to avoid models splitting ? CDO is an impressive technology which have been around in Eclipse for a few years now. It keeps getting better and provides, as a model repository, every service you might dream of. Using it, one is able to build a solution based on pessimistic locking at the model element level, with live updates when you are connected.
	If you are more interested by the optimistic strategy, CDO offers the possibility to provide conflict resolver, to solve conflicts in the more automatic way, when they happend. CDO does not enable only efficient collaborative strategies, it solves the scalabilty issue, by loading only necessary model elements and unloading them automatically.
	I would like to share with you what is happening on a broader scale. Those technologies are only a start in collaborative modeling, and there are many ways to get inspiration about collaboration, github or google docs for instance.

Of course a large part of this talk was reserved for demos of the improvements in Mylyn, EMF Compare, EGit, and Dawn, but I have currently no video of them to share.

Collaborative modeling, the new deal

 I already blogged about collaborative modeling but I would like to stand back from the previous post, which was very technical, and introduce the talk I will present with Martin and Benjamin at EclipseCon Europe.

Since several years now the Eclipse modeling platform provides useful tools which have help to democratize the use of modeling technologies. However those tools have raised issues linked to collaboration we start to better address.

To visualize and edit models in a graphical manner diagrams are often used. However  usually diagramming technologies do not scale well and this is problematic as the more the models and their complexity grow the more one wants to work in group.

When you model with other people, one of the difficulty is to understand reasons behind their choices and design. Document them is of course necessary but the main difficulty is to keep documentation accurate across refactorings and changes.

In fact, often the more the time flies by, the more we lost the intention, documentation being seen as a post-design step.





Collaborate means dealing with concurrent accesses. To enable them you have two strategies. One is to prevent conflicts either through models splitting or with locks. But models splitting is not well supported by tools and coarse-grained locks are not pratical.

 

The other strategy is to allows conflict happens and deals with them when they occurs. It looks like a dice game, and depending on the frequency, it could become hard to deal with conflicts.

Interested by the subject ? Come to the talk to discover what are the available technologies to address those concerns, and understand why collaboration matters in modeling.

Modeling 2.0 : Let's collaborate!

A bit of history

4 years ago, when I joined Obeo, I started to work on a new modeling tool dedicated to system engineering. The main focus of this tool was to enable system engineers to model graphically complex systems with different representations, for each speciality, depending on their viewpoint. The tool was associated to a methodology and integrated in a modeling workbench.

Separation of concerns by representations was a first step to scalability of graphical representations, another was to add several mechanisms (layers, filters, hide/reveal) to allow end users to hide some elements. The approach is somehow similar on what Miles is working on.

We took the decision to create a generic runtime with the possibility to specify graphical representations and to drive the tooling by the viewpoints. The approach has similarities with Jan current's work, and I think our work inspired him a little bit.

Since the runtime has been used for safety analysis (diagram screenshot) and several other use cases.

However, although our tool primary concerns was collaborate modeling since its inception, we did not offer directly a solution to the concurrent access problem. In fact, we left it to the Revision Control System (RCS) used by end users. As Alex wrote concurrent access management by RCS have several drawbacks, so we decided to offer the possibility for end users to collaborate more seamlessly.

On the technical side, we decided to use CDO, a technology we had experienced successfully in others projects, and which seems to match the most to our requirements. As our graphical modeling technology is based on GMF Runtime, our global technical task was to integrate GMF and CDO.

GMF and CDO, integration challenges

If you are used to component architecture like the Eclipse platform one, you may think that these two components will plug together quite easily. Unfortunately, this is not the case for various reasons :

• EMF Transactions and CDO Transactions
  GMF Runtime relies on EMF Transaction to manage read and write accesses to models. To simplify this means that one is not allowed to do a modification without encapsulating its change in a recording command. On its side CDO provides its own transaction mechanism, which relies on the database store one. So it is necessary to integrate the two mechanisms. This is one of the features Dawn provides (see those bugzilla entries for further information).

• Transactional model break in GMF Runtime
  GMF Runtime is a bridge between EMF models and GEF. Among others things, it provides a notation model, to store graphical information. The synchronization between the graphical model and the semantic model, is one of the responsibility of edit parts. The choice to split the synchronization responsibility in each edit part enables one to easily override the default behavior for a given edit part. This choice is consistent with the high extensible approach of the framework. However, it has one major drawback, edit parts should be instantiated to update the notation model ! As GMF Runtime edit parts creation is done in an post commit listener, when no models changes are allowed, the runtime have to force the transaction using a special option.
  This behavior prevents one to have triggers for those modifications. If you need to avoid conflicts (see below), you will not be able to acquire lock before commit. In this case it means that one have to extract the synchronization from edit parts and to call it at the right time.
• Avoiding conflicts
  If CDO provides live updates of shared models, it does not prevents conflicts. There is two ways to handle them. The first one is to write your own conflict resolver, which will take a decision when a conflict occurs. The second one is to avoid conflicts through locks. Easier to said than to implement, but that the way we choose.
• Lock integration
  To avoid conflicts, the simplest way is to lock automatically elements before their edition. In practice, one could use the EMF Transaction trigger mechanism to do this. Edit parts need to listen to locks events to deactivate accordingly their edition capabilities and provide visual indicators of the lock. Lastly, to provide logical lock for end-users, one may provide a lock strategy.
• Consistency maintenance on client side
  EMF Transaction enables one to rollback a transaction, before its commit. This is generally done for validation or lock acquisitions purposes. The recording command manages this rollback and will abort properly the transaction if it occurs. However with the abort, the CDO state of the object will change (see bug 312534 for more details). To avoid conflicts a solution is to use CDO save points to restore the object to a clean state if a transaction aborts.
   
• Native mode for GMF Runtime notation models ?
  One of the coolest thing of CDO is its EObject implementation, but to benefit from the scalability it provides, you need to regenerate your metamodel with a CDO dependency. When one works with such CDO model instances, the mode in which one is, is called native. As GMF Runtime notation metamodel depends on the EMF one (don't do this at home), using notation models in native mode is far from being easy. So we worked on a solution which keeps source and binary compatibility with legacy code (see bug 336707 for technical details) and provides notation cdo native models, the code is available here (no builds yet).
     
• Content and cross reference adapters
  These two utilities should not be used as they will break the lazy loading mechanism of CDO. By default GMF runtime initializes one cross reference adapter when configuring the editing domain. You could quite easily override the default behavior. EMF Transaction uses an adapter that walk through its content, but removing it is much harder. 

On our product, the work is still in progress, but we he have implemented solutions for most challenges presented above. During this work we opened several issues :   

• fixed : bug 345719, bug 352191, bug 355915, bug 337870, bug 353167, bug 329753, bug 315570, bug 329869, bug 329752, bug 353076, bug 351393, bug 351921, bug 325436, bug 346378
• not yet fixed : bug 339908, bug 352189, bug 352977, bug 352204, bug 357400, bug 329786

As you may seen, the CDO team has been very responsive, and already provided fixes for most of them. I would like to thank them for their availability and help.

This blog entry present results coming from a work collaboration between Thales and Obeo.
Ce message de blog est issu d'un travail en commun entre Thales et Obeo.