{"id":945,"date":"2019-11-22T15:12:12","date_gmt":"2019-11-22T15:12:12","guid":{"rendered":"https:\/\/www.pyxll.com\/blog\/?p=945"},"modified":"2023-01-05T10:06:06","modified_gmt":"2023-01-05T10:06:06","slug":"a-better-goal-seek","status":"publish","type":"post","link":"https:\/\/www.pyxll.com\/blog\/a-better-goal-seek\/","title":{"rendered":"Multi-Dimensional Optimization: A Better Goal Seek"},"content":{"rendered":"\n<p class=\"has-background has-very-light-gray-background-color\">The code for the examples can be found in the <i>optimization<\/i> folder of our <a href=\"https:\/\/github.com\/pyxll\/pyxll-examples\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">examples repository.<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Improving on Excel&#8217;s Solver with Python<\/h2>\n\n\n\n<p>Excel comes with an <a href=\"https:\/\/support.microsoft.com\/en-us\/office\/load-the-solver-add-in-in-excel-612926fc-d53b-46b4-872c-e24772f078ca?ocmsassetid=hp010021570&amp;correlationid=35329649-f7df-45fc-b694-16b800077c29&amp;ui=en-us&amp;rs=en-us&amp;ad=us\" target=\"_blank\" rel=\"noreferrer noopener\" label=\" (opens in a new tab)\">installable add-in called the solver<\/a>, which is admirably well-suited to general optimization problems. You will find a simple description of the solver\u2019s capabilities in <a href=\"https:\/\/www.excel-easy.com\/data-analysis\/solver.html\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">this series of articles<\/a>. This is a perfectly satisfactory way to solve many relatively simple models. In spreadsheet work the objective function is typically some <a href=\"https:\/\/en.wikipedia.org\/wiki\/Data_model\"><em>model<\/em><\/a> describing real-world objects and relationships between them.<\/p>\n\n\n\n<p>Although versatile, like most packaged software the solver <a href=\"https:\/\/www.solver.com\/excel-solver-solver-could-not-find-feasible-solution-5\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">has its limitations<\/a> (limited choice of strategies, problem size and constraint size limitations being the most frequent issues), and so at some stage you may well find that you would like Excel to optimize models that are currently outside its capabilities.<\/p>\n\n\n\n<p>PyXLL users can deploy Python-based solutions to extend Excel\u2019s optimization abilities in any number of ways, tailored as necessary to their specific problems.<\/p>\n\n\n\n<p>In this article we consider how to make use of the many advanced optimization facilities readily available in the open source <a rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\" href=\"https:\/\/docs.scipy.org\/doc\/scipy\/reference\/optimize.html#module-scipy.optimize\" target=\"_blank\">scipy.optimize<\/a> library. As a glance at that listing clearly shows there are optimization routines available to cope with all kinds of circumstances.<\/p>\n\n\n\n<p>[maxbutton name=&#8221;download&#8221;]<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Optimization principles<\/h3>\n\n\n\n<p>Optimization is very simple in principle. Then again, so is juggling, but the practice turns out to be rather difficult and far less straightforward than it seems in theory.<\/p>\n\n\n\n<p>Any process of optimization requires the finding of a minimum or maximum value for some function (the so-called <em>objective function<\/em>) that produces a scalar output (to avoid ambiguity in maximisation). The difficulty of the problem is primarily determined by the <em>dimensionality<\/em> of the problem: the number of scalar values in the function\u2019s input.<\/p>\n\n\n\n<p>Starting with chosen initial values the optimizer varies the inputs to the objective function to either maximise or minimise its value. A feedback loop directs the changes in the inputs according to the strategy the optimizer uses and the results of earlier evaluations.&nbsp; This relatively simple statement, however, can hide a multitude of complexities.<\/p>\n\n\n\n<p>They arrive in the form of questions such as these:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>What is an efficient strategy for this particular optimization?<\/li><li>What guarantees are there that the selected optimum is indeed optimal?<\/li><li>What constraints can the optimizer accept on the inputs?<\/li><\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Using Python to optimize in Excel<\/h3>\n\n\n\n<p>While the solver can offer a good first handle on an optimization problem, it cannot hope to offer the sophistication of the many different techniques available in the <code>scipy.optimize<\/code> library, which offers specialised methods for particular types of problem, and efficiency improvements when more is known about the function (for example, if we know the function\u2019s slope at a current evaluation point this allows us to make better guesses about more optimal values for the inputs).<\/p>\n\n\n\n<p><a rel=\"noreferrer noopener\" label=\" (opens in a new tab)\" href=\"https:\/\/docs.scipy.org\/doc\/scipy\/tutorial\/optimize.html\" target=\"_blank\">Scipy\u2019s optimization tutorial<\/a> demonstrates several uses of the library in Python, and will repay further study. As those examples make clear, the best solution to a complex optimization problem often requires some study and experimentation. This post gives you the information you will need to do the necessary experimentation.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Creating the Excel objective function<\/h3>\n\n\n\n<p>Before you can make your spreadsheet provide the objective function for your optimization, you need to understand a little about how Excel\u2019s calculation cycle works. Excel records each cell\u2019s <em>precedents<\/em> (the cells on whose value this cell&#8217;s value depends) and <em>dependents<\/em> (the cells whose value depends on this cell\u2019s value). Whenever any cell\u2019s value is changed the value of all its dependent cells must then be recalculated, and then the values of <em>their<\/em> dependent cells, and so on until no calculation work remains.<\/p>\n\n\n\n<p>The objective function in Excel will simply be a formula in a cell that takes one or more inputs, for which we want to find a set of inputs that produce the minimum result.<\/p>\n\n\n\n<p>Let\u2019s start with a very simple Excel model based on a periodic function with two inputs and a single output. Mathematically we can write that as<\/p>\n\n\n\n<p> <em>z = f(x, y) = sin(x)cos(y)<\/em><\/p>\n\n\n\n<p>Here\u2019s a 3-D plot of the function\u2019s behaviour for values of x and y around the origin:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large img-border\"><img decoding=\"async\" width=\"932\" height=\"526\" data-attachment-id=\"975\" data-permalink=\"https:\/\/www.pyxll.com\/blog\/a-better-goal-seek\/capture-1\/\" data-orig-file=\"https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/Capture-1.jpg\" data-orig-size=\"932,526\" data-comments-opened=\"0\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;Tony Roberts&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;1574435462&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"Capture-1\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/Capture-1.jpg\" src=\"https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/Capture-1.jpg\" alt=\"\" class=\"wp-image-975\" srcset=\"https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/Capture-1.jpg 932w, https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/Capture-1-300x169.jpg 300w, https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/Capture-1-768x433.jpg 768w\" sizes=\"(max-width: 932px) 100vw, 932px\" \/><\/figure>\n\n\n\n<p>As you can see, the periodic nature of the function means that the maxima and minima repeat, so the answer you get will depend on where you start looking (the initial values for <em>x<\/em> and <em>y)<\/em>.<\/p>\n\n\n\n<p>In Excel we can calculate F(X,Y) by putting our inputs X and Y in cells C2 and C3 and then the formula (our objective function) <code>=SIN(C2)*COS(C3)<\/code> in cell C4.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img decoding=\"async\" data-attachment-id=\"959\" data-permalink=\"https:\/\/www.pyxll.com\/blog\/a-better-goal-seek\/capture\/\" data-orig-file=\"https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/Capture.jpg\" data-orig-size=\"668,372\" data-comments-opened=\"0\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;Tony Roberts&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;1574431367&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"Capture\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/Capture.jpg\" src=\"https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/Capture.jpg\" alt=\"\" class=\"wp-image-959\" width=\"501\" height=\"279\" srcset=\"https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/Capture.jpg 668w, https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/Capture-300x167.jpg 300w\" sizes=\"(max-width: 501px) 100vw, 501px\" \/><\/figure>\n\n\n\n<p>The techniques in <code>scipy.optimize<\/code> all rely on the optimizer being able to call an objective function written in Python, so when using an Excel model we need to wrap it to make it possible for the optimizer to call. In pseudo-code terms we might define such a function as follows (remember that in numpy, variables can be multi-dimensional, so <em>x<\/em> might be a vector that is stored in multiple cells):<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\ndef objective_function(x):\n \u00a0\u00a0\u00a0# Set model&#039;s input(s) to x\n \u00a0\u00a0\u00a0# Have Excel calculate the model\n \u00a0\u00a0\u00a0# Return the value of the output cell\n<\/pre><\/div>\n\n\n<p>A simple implementation of an objective function that gets the inputs from cells C2 and C3, calculates the sheet, and reads the result from C4 is as follows: <\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\nfrom pyxll import xl_app\n\ndef objective_func(arg):\n    xl = xl_app()\n    # Copy argument values to input range\n    xl.Range(&#039;C2&#039;).Value = float(arg&#x5B;0])\n    xl.Range(&#039;C3&#039;).Value = float(arg&#x5B;1])\n\n    # Calculate after changing the inputs\n    xl.Calculate()\n\n    # Return the value of the output cell\n    result = xl.Range(&quot;C4&quot;).Value\n    \n    return result\n<\/pre><\/div>\n\n\n<p>Each time the function is called (with a two-element numpy array as its argument) the elements of the array are copied into the input cells, Excel is told to run a calculation cycle and the value of the result cell is returned as the value of the function call.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Solving for X and Y<\/h2>\n\n\n\n<p>Now that we have an objective function <em>objective_func<\/em> we can use <code>scipy.optimize.minimize<\/code> to solve for its minimal value. Using the Nelder-Mead algorithm a suitable line of Python would read:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\nfrom scipy.optimize import minimize\n\nminimize(objective_func, INITIAL_VALUES, method=&#039;nelder-mead&#039;)\n<\/pre><\/div>\n\n\n<p>Here INITIAL_VALUES is the initial value for the function\u2019s arguments. For multivariate optimization problems this should be a simple vector of type <code>numpy.array<\/code>. <\/p>\n\n\n\n<p>To call this from Excel we can use PyXLL&#8217;s <a href=\"https:\/\/www.pyxll.com\/docs\/userguide\/macros.html\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">@xl_macro<\/a> decorator to create an Excel macro that will call the scipy optimizer. This macro can be assigned to a button in Excel, or called via a shortcut key. We could equally well use <a href=\"https:\/\/www.pyxll.com\/docs\/userguide\/menus.html\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">@xl_menu<\/a> or use a <a href=\"https:\/\/www.pyxll.com\/docs\/userguide\/ribbon.html\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\" (opens in a new tab)\">ribbon function<\/a> if that suited our requirements better.<\/p>\n\n\n\n<p>The macro collects the initial values from cells C2 and C3 in the sheet and calls <code>scipy.optimize.minimize<\/code> with our objective function. It also sets Excel to &#8220;Manual Calc&#8221; mode and disables screen updating while the optimization is running. Disabling screen updating is advisable for any task that does lots of updating, as refreshing the screen on each update slows things down. &#8220;Manual Calc&#8221; mode ensures that we can make multiple updates to Excel and only trigger a recalculation once all the inputs have been set. A try\/finally block is used to ensure that Excel&#8217;s settings are restored before the function returns.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\nfrom pyxll import xl_macro, xl_app\nfrom win32com.client import constants\n\n@xl_macro(shortcut=&#039;Ctrl+Alt+P&#039;)\ndef optimize():\n \u00a0\u00a0\u00a0xl = xl_app() \u00a0\u00a0\u00a0\u00a0# Get the initial values of the input cells\n \u00a0\u00a0\u00a0x = xl.Range(&quot;C2&quot;).Value\n \u00a0\u00a0\u00a0y = xl.Range(&quot;C3&quot;).Value\n \u00a0\u00a0\u00a0X = np.array(&#x5B;x, y])\n \u00a0\u00a0\u00a0orig_calc_mode = xl.Calculation\n \u00a0\u00a0\u00a0try:\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# switch Excel to manual calculation\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# and disable screen updating\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0xl.Calculation = constants.xlManual\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0xl.ScreenUpdating = False\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# run the minimization routine\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0minimize(obj_func, X, method=&#039;nelder-mead&#039;)\n \u00a0\u00a0\u00a0finally:\n        # restore the original calculation\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # and screen updating mode\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 xl.ScreenUpdating = True \n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 xl.Calculation = orig_calc_mode\n<\/pre><\/div>\n\n\n<p>Calling this macro via the shortcut key or by assigning the macro to a button solves X and Y in cells C2 and C3 to give the minimum value of F(X,Y) in cell C4.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img decoding=\"async\" data-attachment-id=\"963\" data-permalink=\"https:\/\/www.pyxll.com\/blog\/a-better-goal-seek\/cnu2ykruxn-2\/\" data-orig-file=\"https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/CnU2YkrUXn-1.gif\" data-orig-size=\"691,428\" data-comments-opened=\"0\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"CnU2YkrUXn\" data-image-description=\"\" data-image-caption=\"\" data-large-file=\"https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/CnU2YkrUXn-1.gif\" src=\"https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/CnU2YkrUXn-1.gif\" alt=\"scipy-optimizer-in-excel\" class=\"wp-image-963\" width=\"518\" height=\"321\"\/><figcaption>The solving has been slowed down and screen updates enabled for visualisation purposes<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Further Improvements<\/h2>\n\n\n\n<p>The code as presented so far can be found in file <em>optimize1.py<\/em> in the Github repo. For larger multivariate problems it\u2019s inconvenient to write a separate statement for each value in the function input. In the <em>optimize2.py<\/em> file this inputs are gathered by statements that will cope with a range of any size.<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\n# Get the initial values of the input cells\nin_values = xl.Range(&#039;C11:C12&#039;).Value\nX = np.array(&#x5B;x&#x5B;0] for x in in_values])\n<\/pre><\/div>\n\n\n<p>The <em>optimize3.py<\/em> file takes things a little further, using a named input range and also avoiding the need for the objective function to look up the Excel app each time it is called by making the app a parameter to the objective function and then binding it permanently using <code>functools.partial<\/code>. The objective function declaration is modified thus:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: python; title: ; notranslate\" title=\"\">\ndef obj_func(xl, arg):\n\u00a0\u00a0\u00a0\u00a0&quot;&quot;&quot;Wraps a spreadsheet computation as a Python function.&quot;&quot;&quot;\n\u00a0\u00a0\u00a0\u00a0# Copy argument values to input range\n\u00a0\u00a0\u00a0\u00a0xl.Range(&#039;Inputs&#039;).Value = &#x5B;(float(x), ) for x in arg]\n\n\u00a0\u00a0\u00a0\u00a0# Calculate after changing the inputs\n\u00a0\u00a0\u00a0\u00a0xl.Calculate()\n\n\u00a0\u00a0\u00a0\u00a0# Return the value of the output cell\n\u00a0\u00a0\u00a0\u00a0result = xl.Range(&quot;Output&quot;).Value\n\u00a0\u00a0\u00a0\u00a0return result\n<\/pre><\/div>\n\n\n<p>The optimizer is modified to bind the <code>xl<\/code> argument, leaving it as the required single-argument function :<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: plain; title: ; notranslate\" title=\"\">\nxl_obj_func = partial(obj_func, xl)\nminimize(xl_obj_func, X, method=&#039;nelder-mead&#039;)\n<\/pre><\/div>\n\n\n<h2 class=\"wp-block-heading\">Conclusions<\/h2>\n\n\n\n<p>While the objective function demonstrated in this article is trivial, the same principles apply to any workbook where an input vector is transformed into a scalar objective measure. This allows you to broaden the scope of your optimization efforts while retaining the well-known Excel interface and all its corporate benefits.<\/p>\n\n\n\n<p>For a more flexible solver, it would be an advantage to be able to select the inputs and output via a user interface. This can be achieved using one of Python&#8217;s UI toolkits such as PyQt5 but that is beyond the scope of this article.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Using SciPy in an Excel macro for complex optimization problems.<\/p>\n","protected":false},"author":803,"featured_media":963,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[12,2],"tags":[],"class_list":["post-945","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-python","category-pyxll"],"acf":[],"jetpack_featured_media_url":"https:\/\/www.pyxll.com\/blog\/wp-content\/uploads\/2019\/11\/CnU2YkrUXn-1.gif","jetpack_shortlink":"https:\/\/wp.me\/p7l3LP-ff","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.pyxll.com\/blog\/wp-json\/wp\/v2\/posts\/945","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pyxll.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pyxll.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pyxll.com\/blog\/wp-json\/wp\/v2\/users\/803"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pyxll.com\/blog\/wp-json\/wp\/v2\/comments?post=945"}],"version-history":[{"count":1,"href":"https:\/\/www.pyxll.com\/blog\/wp-json\/wp\/v2\/posts\/945\/revisions"}],"predecessor-version":[{"id":1678,"href":"https:\/\/www.pyxll.com\/blog\/wp-json\/wp\/v2\/posts\/945\/revisions\/1678"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.pyxll.com\/blog\/wp-json\/wp\/v2\/media\/963"}],"wp:attachment":[{"href":"https:\/\/www.pyxll.com\/blog\/wp-json\/wp\/v2\/media?parent=945"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pyxll.com\/blog\/wp-json\/wp\/v2\/categories?post=945"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pyxll.com\/blog\/wp-json\/wp\/v2\/tags?post=945"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}