VPP master, commit: 0f09a47f10e29cabeb98f571e5c4f0c18a54564d

[Reproducible] Running VPP in the background and using VCL to wrap a sample application throws many different errors (many times, all of the errors below).

Sample program: using epoll to listen on a single socket, accepting all connections (and adding them to epoll). Acts as a an ECHO server (messages of constant size of 1024 bytes) with reused connections.

Running with a small load (around 100 connections) works well.

Running with around 1000+ connections shows some of the errors, only a couple of times.

Running with 35K connections throws the following errors:

In VPP:

• app_send_io_evt_rx:587: evt q rings full
• mq_try_lock_and_alloc_msg:167: failed to alloc msg
• app_enqueue_evt:542: evt q full
• svm_msg_q_free_msg:184: message out of order
• only once at end, sometimes:
  • Aborted
  • and the VPP crashes...
• sometimes, after a failed run:
  • Segmentation fault
  • and the only way to start the application (that I found is using a reboot)

In the wrapped application:

• svm_msg_q_free_msg:184: message out of order
• ssvm_delete_shm:211: unlink segment '2562-55': No such file or directory (errno 2)
• vcl_session_accepted:552: vcl<2513:0>: session overlap handle 4294978441 state 4!
• only once at end, sometimes:
  • vl_client_disconnect:331: queue drain: 585

Notes:

1. The sample application is run with a single thread (of a single process).
2. All connections are made at the beginning (non-blocking, one a time though) and are used throughout the test.
3. Trying to run the program multi-process (bind->fork and also fork->bind) or multi-threaded (pthread_create) fails (where obviously running it with POSIX API works great with very high loads).