[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/a/eee/proeco/v267y2024ics0925527323003201.html
   My bibliography  Save this article

Deep Reinforcement Learning for One-Warehouse Multi-Retailer inventory management

Author

Listed:
  • Kaynov, Illya
  • van Knippenberg, Marijn
  • Menkovski, Vlado
  • van Breemen, Albert
  • van Jaarsveld, Willem
Abstract
The One-Warehouse Multi-Retailer (OWMR) system is the prototypical distribution and inventory system. Many OWMR variants exist, e.g. demand in excess of supply may be completely back-ordered, partially back-ordered, or lost. Prior research has focused on the study of heuristic reordering policies such as echelon base-stock levels coupled with heuristic allocation policies. Constructing well-performing policies is time-consuming and must be redone for every problem variant. By contrast, Deep Reinforcement Learning (DRL) is a general purpose technique for sequential decision making that has yielded good results for various challenging inventory systems. However, applying DRL to OWMR problems is nontrivial, since allocation involves setting a quantity for each retailer: The number of possible allocations grows exponentially in the number of retailers. Since each action is typically associated with a neural network output node, this renders standard DRL techniques intractable. Our proposed DRL algorithm instead inferences a multi-discrete action distribution which has output nodes that grow linearly in the number of retailers. Moreover, when total retailer orders exceed the available warehouse inventory, we propose a random rationing policy that substantially improves the ability of standard DRL algorithms to train good policies because it promotes the learning of feasible retailer order quantities. The resulting algorithm outperforms general-purpose benchmark policies by ∼1−3% for the lost sales case and by ∼12−20% for the partial back-ordering case. For complete back-ordering, the algorithm cannot consistently outperform the benchmark.

Suggested Citation

  • Kaynov, Illya & van Knippenberg, Marijn & Menkovski, Vlado & van Breemen, Albert & van Jaarsveld, Willem, 2024. "Deep Reinforcement Learning for One-Warehouse Multi-Retailer inventory management," International Journal of Production Economics, Elsevier, vol. 267(C).
    • Handle: RePEc:eee:proeco:v:267:y:2024:i:c:s0925527323003201
    DOI: 10.1016/j.ijpe.2023.109088
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0925527323003201
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijpe.2023.109088?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Gayon, J.-P. & Massonnet, G. & Rapine, C. & Stauffer, G., 2016. "Constant approximation algorithms for the one warehouse multiple retailers problem with backlog or lost-sales," European Journal of Operational Research, Elsevier, vol. 250(1), pages 155-163.
    2. Woonghee Tim Huh & Ganesh Janakiraman & Mahesh Nagarajan, 2016. "Capacitated Multiechelon Inventory Systems: Policies and Bounds," Manufacturing & Service Operations Management, INFORMS, vol. 18(4), pages 570-584, October.
    3. de Kok, Ton & Grob, Christopher & Laumanns, Marco & Minner, Stefan & Rambau, Jörg & Schade, Konrad, 2018. "A typology and literature review on stochastic multi-echelon inventory models," European Journal of Operational Research, Elsevier, vol. 269(3), pages 955-983.
    4. Tan Wang & L. Jeff Hong, 2023. "Large-Scale Inventory Optimization: A Recurrent Neural Networks–Inspired Simulation Approach," INFORMS Journal on Computing, INFORMS, vol. 35(1), pages 196-215, January.
    5. Sven Axsäter & Johan Marklund, 2008. "Optimal Position-Based Warehouse Ordering in Divergent Two-Echelon Inventory Systems," Operations Research, INFORMS, vol. 56(4), pages 976-991, August.
    6. Johan Marklund, 2011. "Inventory control in divergent supply chains with time‐based dispatching and shipment consolidation," Naval Research Logistics (NRL), John Wiley & Sons, vol. 58(1), pages 59-71, February.
    7. Berling, Peter & Johansson, Lina & Marklund, Johan, 2023. "Controlling inventories in omni/multi-channel distribution systems with variable customer order-sizes," Omega, Elsevier, vol. 114(C).
    8. Diks, E. B. & de Kok, A. G., 1998. "Optimal control of a divergent multi-echelon inventory system," European Journal of Operational Research, Elsevier, vol. 111(1), pages 75-97, November.
    9. Johan Marklund & Kaj Rosling, 2012. "Lower Bounds and Heuristics for Supply Chain Stock Allocation," Operations Research, INFORMS, vol. 60(1), pages 92-105, February.
    10. Andrew J. Clark & Herbert Scarf, 2004. "Optimal Policies for a Multi-Echelon Inventory Problem," Management Science, INFORMS, vol. 50(12_supple), pages 1782-1790, December.
    11. Arthur M. Geoffrion, 1970. "Elements of Large-Scale Mathematical Programming Part I: Concepts," Management Science, INFORMS, vol. 16(11), pages 652-675, July.
    12. Mila Nambiar & David Simchi‐Levi & He Wang, 2021. "Dynamic Inventory Allocation with Demand Learning for Seasonal Goods," Production and Operations Management, Production and Operations Management Society, vol. 30(3), pages 750-765, March.
    13. Kaj Rosling, 1989. "Optimal Inventory Policies for Assembly Systems Under Random Demands," Operations Research, INFORMS, vol. 37(4), pages 565-579, August.
    14. Junxuan Li & Alejandro Toriello & He Wang & Seth Borin & Christina Gallarno, 2021. "Dynamic Inventory Allocation for Seasonal Merchandise at Dillard’s," Interfaces, INFORMS, vol. 51(4), pages 297-311, July.
    15. van der Heijden, M. C. & Diks, E. B. & de Kok, A. G., 1997. "Stock allocation in general multi-echelon distribution systems with (R, S) order-up-to-policies," International Journal of Production Economics, Elsevier, vol. 49(2), pages 157-174, April.
    16. Steven Nahmias & Stephen A. Smith, 1994. "Optimizing Inventory Levels in a Two-Echelon Retailer System with Partial Lost Sales," Management Science, INFORMS, vol. 40(5), pages 582-596, May.
    17. Stenius, Olof & Marklund, Johan & Axsäter, Sven, 2018. "Sustainable multi-echelon inventory control with shipment consolidation and volume dependent freight costs," European Journal of Operational Research, Elsevier, vol. 267(3), pages 904-916.
    18. Bijvank, Marco & Johansen, Søren Glud, 2012. "Periodic review lost-sales inventory models with compound Poisson demand and constant lead times of any length," European Journal of Operational Research, Elsevier, vol. 220(1), pages 106-114.
    19. Paul Zipkin, 2008. "Old and New Methods for Lost-Sales Inventory Systems," Operations Research, INFORMS, vol. 56(5), pages 1256-1263, October.
    20. Olof Stenius & Ayşe Gönül Karaarslan & Johan Marklund & A. G. de Kok, 2016. "Exact Analysis of Divergent Inventory Systems with Time-Based Shipment Consolidation and Compound Poisson Demand," Operations Research, INFORMS, vol. 64(4), pages 906-921, August.
    21. Sven Axsäter & Johan Marklund & Edward A. Silver, 2002. "Heuristic Methods for Centralized Control of One-Warehouse, N-Retailer Inventory Systems," Manufacturing & Service Operations Management, INFORMS, vol. 4(1), pages 75-97, October.
    22. David A. Goldberg & Dmitriy A. Katz-Rogozhnikov & Yingdong Lu & Mayank Sharma & Mark S. Squillante, 2016. "Asymptotic Optimality of Constant-Order Policies for Lost Sales Inventory Models with Large Lead Times," Mathematics of Operations Research, INFORMS, vol. 41(3), pages 898-913, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Fadaki, Masih & Asadikia, Atie, 2024. "Augmenting Monte Carlo Tree Search for managing service level agreements," International Journal of Production Economics, Elsevier, vol. 271(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. de Kok, Ton & Grob, Christopher & Laumanns, Marco & Minner, Stefan & Rambau, Jörg & Schade, Konrad, 2018. "A typology and literature review on stochastic multi-echelon inventory models," European Journal of Operational Research, Elsevier, vol. 269(3), pages 955-983.
    2. Olof Stenius & Ayşe Gönül Karaarslan & Johan Marklund & A. G. de Kok, 2016. "Exact Analysis of Divergent Inventory Systems with Time-Based Shipment Consolidation and Compound Poisson Demand," Operations Research, INFORMS, vol. 64(4), pages 906-921, August.
    3. Dai, Bo & Chen, Haoxun & Li, Yuan & Zhang, Yidong & Wang, Xiaoqing & Deng, Yuming, 2023. "An alternating direction method of multipliers for optimizing (s, S) policies in a distribution system with joint replenishment volume constraints," Omega, Elsevier, vol. 116(C).
    4. Huaxiao Shen & Tian Tian & Han Zhu, 2019. "A Two-Echelon Inventory System with a Minimum Order Quantity Requirement," Sustainability, MDPI, vol. 11(18), pages 1-22, September.
    5. Wang, Qinan & Wan, Guangyu, 2023. "Fixed-interval order-up-to policies and myopic optimal warehouse stock allocation for one-warehouse multiple-retailer systems," European Journal of Operational Research, Elsevier, vol. 309(3), pages 1112-1124.
    6. Andersson, Jonas & Malmberg, Filip & Marklund, Johan, 2023. "Exact analysis of One-Warehouse-Multiple-Retailer inventory systems with quantity restricted deliveries," European Journal of Operational Research, Elsevier, vol. 309(3), pages 1161-1172.
    7. Berling, Peter & Marklund, Johan, 2013. "A model for heuristic coordination of real life distribution inventory systems with lumpy demand," European Journal of Operational Research, Elsevier, vol. 230(3), pages 515-526.
    8. Johan Marklund & Kaj Rosling, 2012. "Lower Bounds and Heuristics for Supply Chain Stock Allocation," Operations Research, INFORMS, vol. 60(1), pages 92-105, February.
    9. Stenius, Olof & Marklund, Johan & Axsäter, Sven, 2018. "Sustainable multi-echelon inventory control with shipment consolidation and volume dependent freight costs," European Journal of Operational Research, Elsevier, vol. 267(3), pages 904-916.
    10. Howard, Christian & Marklund, Johan, 2011. "Evaluation of stock allocation policies in a divergent inventory system with shipment consolidation," European Journal of Operational Research, Elsevier, vol. 211(2), pages 298-309, June.
    11. van der Heijden, Matthieu, 2000. "Near cost-optimal inventory control policies for divergent networks under fill rate constraints," International Journal of Production Economics, Elsevier, vol. 63(2), pages 161-179, January.
    12. Dimitrova, Dimitrina S. & Ignatov, Zvetan G. & Kaishev, Vladimir K. & Tan, Senren, 2020. "On double-boundary non-crossing probability for a class of compound processes with applications," European Journal of Operational Research, Elsevier, vol. 282(2), pages 602-613.
    13. Rau, Hsin & Wu, Mei-Ying & Wee, Hui-Ming, 2003. "Integrated inventory model for deteriorating items under a multi-echelon supply chain environment," International Journal of Production Economics, Elsevier, vol. 86(2), pages 155-168, November.
    14. Kanchanasuntorn, Kanchana & Techanitisawad, Anulark, 2006. "An approximate periodic model for fixed-life perishable products in a two-echelon inventory-distribution system," International Journal of Production Economics, Elsevier, vol. 100(1), pages 101-115, March.
    15. Saha, Kushal & Bhattacharya, Subir, 2021. "‘Buy online and pick up in-store’: Implications for the store inventory," European Journal of Operational Research, Elsevier, vol. 294(3), pages 906-921.
    16. Sonntag, Danja R. & Schrotenboer, Albert H. & Kiesmüller, Gudrun P., 2023. "Stochastic inventory routing with time-based shipment consolidation," European Journal of Operational Research, Elsevier, vol. 306(3), pages 1186-1201.
    17. Zümbül Atan & Ton de Kok & Nico P. Dellaert & Richard van Boxel & Fred Janssen, 2016. "Setting Planned Leadtimes in Customer-Order-Driven Assembly Systems," Manufacturing & Service Operations Management, INFORMS, vol. 18(1), pages 122-140, February.
    18. Linwei Xin, 2021. "Technical Note—Understanding the Performance of Capped Base-Stock Policies in Lost-Sales Inventory Models," Operations Research, INFORMS, vol. 69(1), pages 61-70, January.
    19. Jana Ralfs & Gudrun P. Kiesmüller, 2022. "Inventory management with advance demand information and flexible shipment consolidation," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 44(4), pages 1009-1044, December.
    20. Sven Axsäter & Johan Marklund & Edward A. Silver, 2002. "Heuristic Methods for Centralized Control of One-Warehouse, N-Retailer Inventory Systems," Manufacturing & Service Operations Management, INFORMS, vol. 4(1), pages 75-97, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:proeco:v:267:y:2024:i:c:s0925527323003201. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijpe .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.