[jboss-user] [jBPM] - Session and thread safety

[Marcin Gałązka]

Marcin Gałązka [https://community.jboss.org/people/marcin.galazka] created the discussion

"Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/723455#723455

--------------------------------------------------------------
Hi

jBPM 5.2, PostgreSQL 9.1.

Kris wrote ( https://community.jboss.org/message/613360#613360#613360 https://community.jboss.org/message/613360#613360) that session is thread safe. Well, getWorkItemManager() present in various implementations (e.g. CommandBasedStatefulKnowledgeSession) does not look particulary safe.

Anyway, my biggest issue is with SingleSessionCommandService. It uses *single* EntityManager instance to perform a database-related operations - see JpaPersistenceContextManaager#getApplicationScopedPersistenceContext, it returns the same (wrapped) appScopedEntityManager. Now, there are two scenarios wrt how SingleSessionCommandService#execute() can be called:
- there is no active JTA transaction,
- there is an active JTA transaction (e.g. the method is called from a business layer where someone already started a JTA transaction).

First one is not interesting - execute() will create a new transaction, do its work and commit the transaction. So far so good.

Second one means that in case of many threads calling SingleSessionCommandService#execute() at the same time the same connection will be used to perform many transactions *at the same time*. Now you can pick your favourite exception:
- org.postgresql.xa.PGXAException: Transaction interleaving not implemented (see http://jdbc.postgresql.org/documentation/faq.html#transaction-interleaving http://jdbc.postgresql.org/documentation/faq.html#transaction-interleaving)
- org.hibernate.AssertionFailure - an assertion failure occured (this may indicate a bug in Hibernate, but is more likely due to unsafe use of the session)
- org.postgresql.xa.PGXAException: Connection is busy with another transaction

You can say that SingleSessionCommandService#execute() is synchronized but it does not help here. The commit (and the flush from EntityManager) happens elsewhere. Overall I think that SingleSessionCommandService is playing a very dangerous game in JTA managed environment. What is the rationale for *SingleSession*CommandService and PersistenceContextManager#getApplicationScopedPersistenceContext() anyway?

A simple test case attached, run few times or increase totalWorkers if you do not get exception at the first pass. If you want more standalone (e.g. Maven based) example let me know. A Spring-based simple web application can be provided as well.
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/723455#723455]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Mauricio Salatino]

Mauricio Salatino [https://community.jboss.org/people/salaboy21] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/723456#723456

--------------------------------------------------------------
Marcin, first of all great initiative!
Please provide a maven project with some tests around this issue and I will personally take a look on that. If we can have a project with the postgre dependencies we can also check that everything works well in different databases. If you add the spring configurations and the different options it will be great.
If you want to contact me privately to do a follow up of the progress and the analyses feel free to drop me an email, but I suggest you to keep it here in the community forums so everybody can leverage our work and fixes. :)
Cheers
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/723456#723456]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Marcin Gałązka]

Marcin Gałązka [https://community.jboss.org/people/marcin.galazka] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/723534#723534

--------------------------------------------------------------
Test case attached at first post (workflow-test-case.zip). Run mvn clean test and wait some time. Test will pass if you remove transaction from ThreadSaferyTest#startProcessTest. This way SingleSessionCommandService#execute() will start its own transaction instead joining existing.
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/723534#723534]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Mauricio Salatino]

Mauricio Salatino [https://community.jboss.org/people/salaboy21] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/723569#723569

--------------------------------------------------------------
Cool, I will start looking at it tomorrow. I will try it with mysql and then I will install postgre to see if the behavior looks similar.
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/723569#723569]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Marcin Gałązka]

Marcin Gałązka [https://community.jboss.org/people/marcin.galazka] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/723636#723636

--------------------------------------------------------------
New version (workflow-test-case-v2.tar.gz) attached. Use 'mvn -Dgroups=spring clean test' to run Spring based test or 'mvn -Dgroups=simple clean test' to run non-Spring based test. Do not run both tests at once ('mvn clean test') - it looks that due to the some background thread spawned by Bitronix TM a data source isn't unbinded from JNDI fast enough and you will get 'java.lang.IllegalArgumentException: resource with uniqueName 'jdbc/jbpm-ds' has already been registered'. I did not investigate the matter further, it is irrelevant here.

I've also fixed parallel tests (mis)configuration.

Every tests (Spring, non-Spring) contains two methods:
- testOwnTransaction - should pass.
- testJoinedTransactions - should fail.
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/723636#723636]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Marcin Gałązka]

Marcin Gałązka [https://community.jboss.org/people/marcin.galazka] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/724932#724932

--------------------------------------------------------------
Any news?
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/724932#724932]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Sebastian Calbaza]

Sebastian Calbaza [https://community.jboss.org/people/calbazasebastian] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/757722#757722

--------------------------------------------------------------
I'm having the same issue... JTA managed transactions with a single statefull session that is accesed concurrently...
Because  SingleSessionCommandService uses a single instance of JpaPersistenceContextManager that returns the same EntityManager s will result in inconsistent results when it's accesed concurrently. My workaround was to implement a thread safe ProcessPersistenceContextManager and set it in the drools Environment when creating the session. ( env.set(EnvironmentName.PERSISTENCE_CONTEXT_MANAGER, persistenceContextManager));.
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/757722#757722]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Gábor Farkas]

Gábor Farkas [https://community.jboss.org/people/hhcofcmds] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/759641#759641

--------------------------------------------------------------
The same problem here, still exists in JBpm5, 5.3.0.Final
The problem is that SingleSessionCommandService calls beginCommandScopedEntityManager on the PersistenceContextManager, but the endCommandScopedEntityManager in called when the transaction commits. When used in an EE environment, the JTA transaction commits well after the SingleSessionCommandService exits the synchronized block.
If another requests comes, and the execution enter the critical section, the CMD_SCOPED_ENTITY_MANAGER is still stored in the environment, so the new request will just use the EntityManager from the previous request, which clearly causes serious problems.
I think that this should be reportes as a bug in Jbpm5, because this prevents the correct usage of JBpm in an enterprise container. What do you think?
In my workaround, I created a CDI interceptor that synchronizes methods that use KnowledgeSession on a higher level.
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/759641#759641]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Rahul Agrawal]

Rahul Agrawal [https://community.jboss.org/people/rahulamt] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/759803#759803

--------------------------------------------------------------
Hi Marcin,

Any updates on this issue?

I think the issue I was facing, as been described here is https://community.jboss.org/thread/204460 https://community.jboss.org/thread/204460 , is because of SingleSessionCommandService not being thread safe.
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/759803#759803]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[wqe wqe]

wqe wqe [https://community.jboss.org/people/drupalspring] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/759864#759864

--------------------------------------------------------------
Oh , it seem it is a critical issue , but i feel so sad no official updates on this issue so far  .Can anyone clarify that this issue still exsit or not , or already been fixed?
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/759864#759864]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Sebastian Calbaza]

Sebastian Calbaza [https://community.jboss.org/people/calbazasebastian] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/760010#760010

--------------------------------------------------------------
Here is my thread safe ProcessPersistenceContextManager if someone needs it.
It needs to be set on the Environment param (env.set(EnvironmentName.PERSISTENCE_CONTEXT_MANAGER, persistenceContextManager);) before you  pass it to JPAKnowledgeService.newStatefulKnowledgeSession(kbase,null, env)

-----------------------------------------
public class MyJpaProcessPersistenceContextManager implements
        ProcessPersistenceContextManager, PersistenceContextManager {

    private EntityManagerFactory emf;

    private ThreadLocal<EntityManager> appScopedEntityManager=new ThreadLocal<EntityManager>();
    protected ThreadLocal<EntityManager> cmdScopedEntityManager=new ThreadLocal<EntityManager>();

    public ProcessPersistenceContext getProcessPersistenceContext() {
        return new JpaProcessPersistenceContext(cmdScopedEntityManager.get());
    }

    public MyJpaProcessPersistenceContextManager(Environment env) {
        this.emf = (EntityManagerFactory) env.get(EnvironmentName.ENTITY_MANAGER_FACTORY);
    }

    public PersistenceContext getApplicationScopedPersistenceContext() {
        if (this.appScopedEntityManager.get() == null) {
                this.appScopedEntityManager.set(this.emf.createEntityManager());
        }
        return new JpaPersistenceContext(appScopedEntityManager.get());
    }

    public PersistenceContext getCommandScopedPersistenceContext() {
        return new JpaPersistenceContext(this.cmdScopedEntityManager.get());
    }

    public void beginCommandScopedEntityManager() {
        if (cmdScopedEntityManager.get() == null ||
                (this.cmdScopedEntityManager.get() != null && !this.cmdScopedEntityManager.get().isOpen())) {
            this.cmdScopedEntityManager.set( this.emf.createEntityManager());
        }
        cmdScopedEntityManager.get().joinTransaction();
        appScopedEntityManager.get().joinTransaction();
    }

    public void endCommandScopedEntityManager() {
        if (this.cmdScopedEntityManager.get()!=null){
            this.cmdScopedEntityManager.get().flush();
            this.cmdScopedEntityManager.get().close();
        }
    }

    public ThreadLocal<EntityManager> getCmdScopedEntityManager() {
        return cmdScopedEntityManager;
    }

    public ThreadLocal<EntityManager> getAppScopedEntityManager() {
        return appScopedEntityManager;
    }

    public void dispose() {
            if (this.appScopedEntityManager.get() != null && this.appScopedEntityManager.get().isOpen()) {
                this.appScopedEntityManager.get().flush();
                this.appScopedEntityManager.get().close();
            }
            this.appScopedEntityManager.set(null);
            if (this.cmdScopedEntityManager.get() != null && this.cmdScopedEntityManager.get().isOpen()) {
                this.cmdScopedEntityManager.get().flush();
                this.cmdScopedEntityManager.get().close();
            }
            this.cmdScopedEntityManager.set(null);
    }
}
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/760010#760010]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Wooi Joe Chuah]

Wooi Joe Chuah [https://community.jboss.org/people/joe.chuah] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/811194#811194

--------------------------------------------------------------
Hi Sebastian,

Are there any drawbacks that should be noted when using your solution?
Any effects on transactions and the sort? CMT / BMT?

Thanks in advance
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/811194#811194]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Sebastian Calbaza]

Sebastian Calbaza [https://community.jboss.org/people/calbazasebastian] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/760012#760012

--------------------------------------------------------------
As a note.. if you are using JPAWorkingMemoryDbLogger you need to override the  getEntityManager/0 method. Mine is like this:
final JPAWorkingMemoryDbLogger logger = new JPAWorkingMemoryDbLogger(
                ksession) {
            @Override
            protected EntityManager getEntityManager() {
                return persistenceContextManager.getCmdScopedEntityManager().get();
            }
        };
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/760012#760012]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Rahul Agrawal]

Rahul Agrawal [https://community.jboss.org/people/rahulamt] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/761041#761041

--------------------------------------------------------------
One of the solution can be to have as many StatfulKnowledgeSession as process instances. The jbpm document mentions that creating a statefulknowledgesession is lightweight,  it the knowlegebase creation that is heavy. So we can create knowledgebase once and knowledgesession multiple times.
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/761041#761041]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Michał Mański]

Michał Mański [https://community.jboss.org/people/mmanski] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/762354#762354

--------------------------------------------------------------
Hi,
I use JBPM 5.3.0.Final in my project. My application uses EJB 3.1/JPA 2.0 and runs on JBoss 7.1.1.Final. I have similar problem with thread safety.
When I use one session for whole application, strange exceptions are thrown (e.g. "javax.persistence.PersistenceException: error during managed flush ... caused by: org.hibernate.AssertionFailure: collection [null] was not processed by flush()", "javax.persistence.PersistenceException: error during managed flush ... Caused by: java.util.ConcurrentModificationException" or "org.hibernate.HibernateException: Found two representations of same collection: org.jbpm.persistence.processinstance.ProcessInstanceInfo.eventTypes") if many users use the application simultaneously. I tried to use the ProcessPersistenceContextManager presented by Sebastian, but it didn't resolve the problem - errors still appear.

When I try to user another solution and create new session for every request, I don't know how to dispose sessions. I read in documentation that this method must always be called after finishing using the session, or the engine will not free the memory used by the session. However, I don't know how to correctly dispose a session.

If I try to dispose session before end of transaction, an exception is thrown during committing transaction: "javax.persistence.PersistenceException: error during managed flush ... Caused by: java.lang.IllegalStateException: Illegal method call. This session was previously disposed".
On the other hand, if i dispose session after transaction commits, another exception is thrown: "Could not commit session: javax.persistence.TransactionRequiredException: No active JTA transaction on joinTransaction call".

Does anyone know what is the correct way to dispose a session?
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/762354#762354]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Maciej Swiderski]

Maciej Swiderski [https://community.jboss.org/people/swiderski.maciej] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/762573#762573

--------------------------------------------------------------
Guys, I created https://issues.jboss.org/browse/JBPM-3814 jira issue for this so please keep an eye on it for updates. Please put all your comments/ideas/recommendations in jira as well

Cheers
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/762573#762573]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]

[Sebastian Calbaza]

Sebastian Calbaza [https://community.jboss.org/people/calbazasebastian] created the discussion

"Re: Session and thread safety"

To view the discussion, visit: https://community.jboss.org/message/765795#765795

--------------------------------------------------------------
I provided a Pull Request ( https://github.com/droolsjbpm/drools/pull/147 https://github.com/droolsjbpm/drools/pull/147) that extends the lock/protection domain till the JTA transaction is commited.

Sebi
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/765795#765795]

Start a new discussion in jBPM at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2034]