Reinforcement Learning to determine the optimal number of queues

Question

question

Elax Kot asked Mar 13, '25 Elax Kot commented Mar 20, '25

Reinforcement Learning to determine the optimal number of queues

Hello,

I am trying to make a reinforcement learning process in Flexsim. In our system, the processor 1 and 2 is a lot faster than processor 3. When starting processor 1 or 2, the system ensures that at least one queue is empty. If all of the queues are occupied, then the item will be held before processor 1/2. I want to determine if there is a way for me to find out what is the required number of queues. This is a simplification of the process that I am trying to optimize. Preferably, I want to use reinforcement learning.

In addition, since each queue has a specific logic (allowing only certain kinds of items to enter, with a certain "weight limit"), is there a way to duplicate the queues without copying the flexscript over and over again? Thanks!

Software Version:

FlexSim 23.2.0

queue reinforcement learning constraints

1741887559827.png (177.6 KiB)

· 2

5 |100000

Attachments: Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Joerg Vogel commented · Mar 14 at 07:20 AM

@Elax Kot, is there a way? Add more 3D processes and check how utilization changes in relation.

A typical concept of object- oriented programming is inheritance.

https://docs.flexsim.com/en/25.1/Using3DObjects/WorkingWith3DObjects/UsingTemplates/UsingTemplates.html#what

You can search in online documentation by keywords. Sometimes you get answers. In this case it would be successful.

1 ·

Elax Kot Joerg Vogel commented · Mar 14 at 08:08 AM

Thanks! I will look into using inheritance with our codes.

0 ·

Answer 1 · 2025-03-14T08:05:47Z

Felix Möhlmann answered Mar 14, '25 Elax Kot commented Mar 20, '25

You can add (equal) queue to a group and link that group to a model parameter. The option "Delete and Copy Group Members" will create as many copies of the first object in the group as the parameter states.

You can then run an experiment to see how many queues you need (for the upstream processors to never be blocked I assume).

Here's a basic example where the number of processors needed to not get any backlog in the queue is determined that way.

experiment-example.fsm

Another approach would be to not change the number of queues and instead assume an unlimited capacity. By measuring the maximum content you can then calculate how many queues would have been needed to fit all material.

While you say that this is a simplification and I am by no means an expert when it comes to Reinforcement Learning, I don't see how it could be applied here. An RL agent is meant to make decisions during the model run based on the state of the model. Varying the starting conditions is what the experimenter/optimizer is for.

experiment-example.fsm (93.2 KiB)

· 3

5 |100000

Attachments: Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Elax Kot commented · Mar 14 at 08:17 AM

Thanks! Using experimenter with range is indeed quite useful and would probably work. An additional constraints that I have would be having to accommodate for the custom code that I have added in the individual queue. Each duplication of the queue will have a slight modifications with regards to the tagging when I use it to interact with a global table that tracks the capacity of the queues.

1. Items that enters a queue will trigger an update to a global table.

2. Items that leaves a queue will also trigger an update.

This is doable when I'm manually copying and editing the custom code in each queue. However, I have no idea how to do it when I'm modifying using the experimenter. Is there a way for me to modify the custom code depending on the queue index?

This would be something that I'm looking to solve as well.

I agree with your point about not using RL in this usecase. It does sound like an overcomplication as opposed to just using experimenter. Many thanks for your response again.

0 ·

Felix Möhlmann Elax Kot commented · Mar 14 at 08:37 AM

If possible, I would make the differences in the code depend on labels on the object. Those can be set in the OnSet code of the parameter.

experiment-example_1.fsm

1 ·

capture1.png (10.0 KiB)

experiment-example-1.fsm (30.1 KiB)

Elax Kot Felix Möhlmann commented · Mar 20 at 09:39 AM

That'll work wonderfully. Thank you so much!

0 ·

question