Bringing Automation to the Datacenter Mike Resseler Veeam Product Strategy Specialist, MVP, Microsoft Certifed IT Professional, MCSA, MCTS, MCP 2 Bringing Automation to the Datacenter 1. NIST defnition of cloud computing: http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf Introduction Being an IT administrator requires a whole different way of working and thinking compared to a couple of years ago. IT professionals used to talk about servers on a day-to-day basisthat time has gone. Today they are spending their time dealing with the growing demand for business services to support end users and customers in an increasingly complex infrastructure. At the same time, IT professionals need to deliver very high quality services and are working under more budget stress than before. There are multiple reasons behind this change. One of the reasons is the Bring Your Own Device (BYOD) principle. Customers and end users are more demanding today than they were a couple of years ago. They are also much more IT aware. The generation entering the workforce now has grown up with digital equipment and has never experienced the sound of an old modem. And they cant imagine a day without Internet. These workers request the best services and they want them anywhere, anytime and on any device. Another reason for this transformation is virtualization. What started a couple of years ago as a cost-saving solution quickly became much more than that. Virtualization offered the possibility of quickly reacting to demands of the business and provisioning services much faster. But virtualization has become an enabler for more. The third reason for the shift in thinking is cloud computinga buzz word for a couple of years now, it is quickly becoming a reality in many organizations. And even if it is not a reality, it will be on the design table for sure. Cloud computing is defned by fve essential characteristics, one of which is automation. While IT organizations are working hard to cope with these new paradigms, they are discovering a lot of challenges: 3 Bringing Automation to the Datacenter Manual processes. IT professionals who have been in the feld for a while know that they used to do everything manually. With the current demands on IT, this is no longer an option. Everything has to be done immediately and is automated. Today business owners and application owners are exploring available cloud services to get their jobs done. Internal IT organizations need to cope with this reality and make sure that they are as fexible as those cloud services. Besides being as fexible, they need to prove their added value compared to cloud services. Underutilization of assets. For years IT professionals bought resources for each specifc project. This led to a lot of hardware never being used fully. Virtualization is a game changer in this area, but too many assets are still over-provisioned and therefore cost more money than they should. Overutilization of assets. The other way around is also seen more frequently. Using techniques such as overcommit or dynamic memory can be a great asset in your environment, but when the pressure becomes too large, it can lead to performance degradation when your services need additional resources. Changes. Virtualization has already brought the possibility of rapidly deploying new assets and services. The private cloud continues on that path and delivers self-service. It becomes even more diffcult to know what is deployed and who is using what in your environment and to have it documented. Always on principle. IT has to be there 24x7 and cannot fail. Because there are better technologies to provide 24x7 uptime, end users and customers expect it and they see it as normal. After all, in their experience, FaceBook is never down and is always available. Complexity. Shared storage, virtual networking, cluster shared volumesall of these new concepts bring in new technologies and add a lot of complexity. And every added layer in your infrastructure can become the failure point. Benefts of datacenter automation Automation in the datacenter brings a lot of advantages and addresses challenges introduced by BYOD, virtualization and cloud computing. Lets look at a few of these advantages: Compliance and standardization Its no secret that people do not perform tasks in the exact same way. It is also not a big secret that any IT professional who has to do a job multiple times 4 Bringing Automation to the Datacenter starts to do it blindly, leading to mistakes. And last but not least, doing tasks manually leads to steps being forgotten. No matter how good you are at creating lists, chances are you will forget that one server in the corner or will miss a rather crucial step in the process. By bringing in automation, you ensure that every time repetitive tasks are performed they are done in the exact same way, which leads to standardization of your environment and compliance. No more forgotten patches (or forgotten servers, for that matter). No more different server deploymentsall are rolled out in the exact same way. Automation will make your life much easier and you will be able to manage your environment better. Server effciency I talked about underutilization (and overutilization) in a white paper I wrote on capacity planning. Having good automation processes in place can help ensure that your VMs are implemented correctly and have the resources they need. With automation, your VMs are adjusted on the fy by monitoring different thresholds when more resources are needed. Order Automation and streamlined processes bring order to the daily chaos you are working in. This revolution has changed industry (e.g., assembly lines in an automobile manufacturing plant) and is now on its way to the IT industry. With reference to the automotive industry, it used to take days before a car was fnished, and after bringing in automation it took less than two hours. Bringing automation to your datacenter will ensure that you can respond faster and in the same way over and over again. Costs Automation simply reduces costs across your IT operations life cycle. Less manual work, fewer human errors, easier and faster troubleshooting because of standardization, fewer incidents because of compliance and less downtime are just some of the cost-reducing factors in your environment. Note: If you are worried about losing your job as a result of automation, then read on to set your mind at ease. Better change management If you are working in a production environment, you will have to deal with change management. I used to spend many hours in CAB (change advisory board) meetings, preparing endless documents on how to handle the change and lag time between every step of a process, and it was sometimes huge. Automation can handle a lot of those issues. 2. A good read on assembly lines can be found here: https://en.wikipedia.org/wiki/Assembly_line 3. CAB: Change Advisory Board: http://en.wikipedia.org/wiki/Change_Advisory_Board 5 Bringing Automation to the Datacenter Better productivity Even if you are not automating every part of a process in your datacenter, automating a part of it can beneft your organization. For example, IT operators have more information after an alert because an automation process has gathered that information. What to automate? One of the major challenges for your automation strategy is the question: what are we going to automate and what are we not going to automate? No, this is not an easy question to answer and you wont fnd a straight answer to this question in this white paper, but here are a few tips to help you resolve this question: Are you or your fellow administrators performing a certain task on a regular basis? If the answer is yes, then all of these tasks can potentially be automated. Do you have issues with keeping your documentation up-to-date? Or are you spending too much time creating and maintaining documentation? If so, automation is an answer to this problem. Have you experienced too much human error with some tasks? Then look into that task and try to automate it. This will eliminate the human error and the resulting diffculties. Are you suffering from inconsistent servers and services? Then you should consider automation. What about known issues in your datacenter? Are errors coming out of your monitoring solution on a frequent basis? This can be automated as well. These questions and others lead to the same point: Will automation save you time and possible issues (and, therefore, money)? For each task you want to automate, ask yourself these questions: How much time will it take to automate? How many hours will it save when the automation is in place? Is it critical? For example, the level of criticality could be assessed by asking whether speed is important for a particular task or if human error could end up costing a fortune. 6 Bringing Automation to the Datacenter How many times will you need to adapt your automation script or workfow in the future? Of course it is diffcult to foresee what will come to your environment but if you are creating automation tasks for a service that changes every six months, then it might not be necessary. Do you need up-to-date documentation based on rules, compliance or another reason? If so, then automating this is a no-brainer. Choosing an automation solution strategy There are a few ways to automate tasks. The most well-known is probably using scripts. You have been doing this for years and probably have a full set of scripts in your environment. Depending who has written the script, it will be in scripting language A or B and will be fully documented or not. There are a number of strategies you can apply today to automate your datacenter. Note that Im not suggesting you throw out everything you have today, but it is wise to decide on an automation strategy that can be applied throughout the entire company and create any new automation task according to that strategy. Older scripts can be changed to the new strategy when they are due for change. Consider your automation strategy from now on as a part of your datacenter vision. Scripting Scripting has evolved over the years and todays standard is PowerShell. Almost every vendor has a specifc PowerShell module for its solution and because of the consistency in language, it is pretty easy to learn this language and create automation. Figure 1: PowerShell window 7 Bringing Automation to the Datacenter If you choose an automation strategy based on scripting, then PowerShell is certainly something to look at. Not only is it fully usable throughout the entire Microsoft stack, but many vendors such as VMware and Citrix have embraced PowerShell as the standard in scripting automation. PowerShell is very easy to learn and there are a lot of good courses out there that will get you started. Here are some important basics to get you started: A commandlet (cmdlet) is a basic instruction. A module is a collection of different commandlets, which are mostly grouped by a specifc technology. PowerShell works with objects, so each commandlet has methods and properties. Pipelines are a series of connected segments, which allow you to use the output (or result) of one commandlet and feed that (input) to another commandlet. Common parameters are very handy to test your scripts, debug them or get more information about what it is doing. Automation software Automation software is very popular in datacenter automation today. There are a few kinds out there, like Citrix Workfow Studio or vCenter Orchestrator, but because both tools are limited when it comes to using them outside of their own solutions, this white paper will only look at Microsoft System Center Orchestrator, which integrates with Microsoft solutions and third-party solutions such as VMware, HP and more. Microsoft System Center Orchestrator Part of the Microsoft System Center suite, Orchestrator is an automation workfow engine that allows you to automate many processes or tasks in your datacenter. There are a few basic terms you need to know: A runbook contains individual activities that make up a workfow. In other words, runbooks contain the instructions for an automated task or process. An activity is an individual step within a runbook. An Integration Pack (IP) is a collection of activities that are related to a specifc technology. For example, the VMware IP is a collection of activities that allow you to do specifc jobs on your VMware infrastructure. 8 Bringing Automation to the Datacenter System Center Orchestrator consists of a few components: The management server is the communication layer between the Runbook Designer and the orchestration database. Your runbooks will continue to run when this server is down but you cant create new ones or change existing ones. A Runbook Server is a server where a runbook runs. Runbook servers communicate directly with the orchestration database. You can deploy multiple Runbook Servers to increase capacity and redundancy but you can also deploy specifc Runbook Servers in an isolated network environment or for specifc purposes. The Orchestrator Database is a Microsoft SQL Server database that contains all of the deployed runbooks, status, log fles and confguration data. Runbook Designer is a tool that allows you to build, edit and manage runbooks. It can be deployed on workstations or management servers. The Runbook Tester allows you to test your runbooks. The Orchestration Console is a website that allows you to start, stop and view real-time status. The web service is a REST-based service that allows applications and services to connect to Orchestrator to start and stop runbooks. It can also allow you to feed information to runbooks and to receive results from those runbooks. Deployment Manager is a tool used to deploy Integration Packs, Runbook servers and Runbook Designers. Figure 2: System Center 2012 Orchestrator Overview 4. REST: Representational State Transfer: http://en.wikipedia.org/wiki/Representational_state_transfer 9 Bringing Automation to the Datacenter To create automation workfows, you use the Runbook Designer (Figure 3). Figure 3: System Center 2012 Orchestrator Runbook Designer Runbook Designer allows you to use the Integration Packs and their activities that are deployed into your environment. When designing your runbooks, you can use a variety of activities to create your workfow. By using Integration Packs you will be able to have more activities that can be used in your workfows. And if you dont have a specifc technology, you can always incorporate scripts into your runbooks. Challenges in automation Automation brings challenges to the table. Here are a few that you might encounter during your automation adventure Help, Ill lose my job One of the biggest misconceptions is that you will lose your job because of automation. After all, when many of your tasks are going to be handled by automation workfows, why are you still necessary? Its true that bringing automation (such as assembly lines) into factories resulted in a reduced need for employees. But automation in the IT industry will not cause you to lose your job. Your responsibilities will change, so you might have to reinvent yourself, but in the end you still will have a job. Instead of being the person who works on the assembly line, you will become the 5. For more information on deploying and registering Integration Packs: http://technet.microsoft.com/en-us/library/hh420346.aspx 10 Bringing Automation to the Datacenter person who invents and builds the assembly line. This will lead to less adhoc frefghting, giving you a more relaxed life, and it will also lead to more challenges. And automation is never fnishedIT transforms so quickly that the need for automation and change will be ongoing. The best advice is to embrace automationit will become your best friend and will give you more than job security. I cant write scriptsIm not a programmer For those who have never written scripts and prefer to work with a graphical user interface, automation sounds intimidating at frst. Today, with the latest version of PowerShell, there is no need to be intimidated by scripting. And with automation tools, you may not be scripting at all. Dont just start automatinghave a strategy In many cases you will see quick wins by automating easy tasks as a starting point. This leads to the belief that you can automate everything that crosses your path. Look back at the What to automate? topic and think about every task before you start automating. Even when you have thought about the different tasks you want to automate, dont just start automating them. Agree on a strategy frst. Read the Best Practices section for more information. If you are going to use scripting languages such as PowerShell, then stick to one (1!) scripting language and let your IT administrators follow a specifc training. If you are going to use an automation solution, make sure that you offer a training for that tool also. When using automation scripts or workfows, people tend to forget the need for logging, error-handling or exception handling. Make sure that you implement mechanisms to catch issues. Finally, do monitor the automation scripts. You can use monitoring tools to look at specifc events or specifc monitoring through System Center 2012 Operations Manager for Orchestrator. Without monitoring, you wont know whether your workfows have run or succeeded, and that can lead to very bad situations. 11 Bringing Automation to the Datacenter Best practices There are several best practices that can point you in the right direction as you are starting your automation adventure. Naming conventions It is always a good idea to agree on naming conventions when you start creating scripts or runbooks. Besides using naming conventions inside scripts, script names and runbooks, it is also a good idea to think about a naming convention when you are using folders in System Center 2012 Orchestrator. Rename your activities Rename your activities in System Center 2012 Orchestrator so they are easy to read. Here is an example: Figure 4: Runbook with standard activity names While you can imagine something with this workfow (and the workfow works), it is unreadable for somebody who hasnt created the workfow. 12 Bringing Automation to the Datacenter Now lets look at the same workfow with changed activity names: Figure 5: Runbook with changed Activity names The second one is much more readable, not only for you, but more importantly for your colleagues too. Built-in error handling Even when you are using automation and workfows there is the possibility that something will go wrong. You will save yourself lots of trouble by building in error and warning handling. Connect errors and warnings to an incident management system (such as Microsoft System Center 2012 Service Manager) or event viewer to get all the information about a failed workfow. Figure 5 illustrates how activities can be used to catch warnings and errors in this runbook. It is a common practice to put warnings on top and errors at the bottom. Use colors Use predefned colors to defne workfows. For example, use green lines when the previous activity has succeeded, orange when you have encountered a warning and red when the activity went wrong. 13 Bringing Automation to the Datacenter Figure 6: Added colors to the runbook Name your links Give specifc names to the links between activities to make it more readable. Figure 7: Adding link labels Use logic Whatever the output from your workfow, use that information in the rest of your workfow to make sure that you do the right activity next. 14 Bringing Automation to the Datacenter Start smallvery small Dont create runbooks that contain lots of activities and a lot of data manipulation and logic. Start with small, easy-to-create runbooks. Its better to create smaller runbooks and link different runbooks together than to create large runbooks, which become unreadable and uneditable after a while. Dont automate your garbage There are too many cases where people try to automate bad workfows that they have been performing manually for years. Review your manual workfow frst, because automating a bad workfow will not magically transform it into a better workfowa bad workfow will remain a bad workfow, even if you automate it. PowerShell scripts Even if you only want to use PowerShell scripts, it is interesting and benefcial to investigate System Center 2012 Orchestrator. By using the Run .NET Script activity, you can add simple and short PowerShell scripts and one-liners and write the output on the data bus. You still will be able to use Orchestrator to make it more readable and use advanced triggering for your scripts. Call runbooks from others Here is an example showing why its a best practice to keep your runbooks small. By using small runbooks and connecting them using the invoke runbook activity, you will be able to use a specifc runbook in multiple fows, making the troubleshooting/updating process much easier. Figure 8: Using invoke activities Another advantage of building smaller runbooks and calling runbooks is that you will be able to reuse already created runbooks. 15 Bringing Automation to the Datacenter Some scenarios Here are several examples of scenarios that are candidates for automation in many organizations. Administrative tasks Think about the tasks that you need to do as an IT administrator, such as user provisioning and deprovisioning, SharePoint site creation, mailbox creation, cleaning of log fle folders and many more. These are the tasks that are performed on a regular basis and are time-consuming (and easily forgotten, too). Many companies start with those scenarios because they are very easy to implement. In many cases, the workfows exist on paper and can be quick wins for the IT department. Having those tasks in place gives you time to start automating other scenarios. Cloud, virtual machine, datastoreprovisioning Every task in your datacenter is up for automationfrom virtual machine (VM) provisioning to a predefned template and advanced workfows that include provisioning storage, network and a VM based on user-requested parameters. This is why its a good idea to start small and divide large workfows into smaller chunks. Disaster recovery planning and testing Bringing automation into your disaster recovery (DR) planning is always a good idea. You can use automation to test your recoveries on a regular basis and you can create workfows to be used when a real disaster strikes. But whenever new assets are created within your infrastructure it is also a good idea to automate the creation of a backup job for those assets or to modify existing backup jobs to include those new (or changed) assets. You can easily add Veeam Backup & Replication into your workfows because of its PowerShell support. Some of the examples that have been built are found later in this document. Event handling, incident and problem management As stated earlier, you can create runbooks or workfows based on specifc events. System Center 2012 Orchestrator includes some activities that can be used to support these kinds of workfows. A few of the available activities are: Monitor Date/Time, Monitor Disk Space, Monitor Event Log, Monitor Internet Application, Monitor Process, Monitor WMI, Monitor Service, Monitor File 16 Bringing Automation to the Datacenter or Folder, Find Text in logfles, Monitor Alert (through Operations Manager), Monitor State of a component, service or Distributed Application (through Operations Manager) and Monitor objects (tickets) that are created in System Center Service Manager. Automation with Veeam and Microsoft System Center This fnal topic looks at what Veeam Software and System Center can do in your automation adventure. These examples are automation workfows that have been built on various occasions by and for customers. They are reference examples that can be used in your own environment. The last example is shown with more details and can give you a head start on how to build automation in your environment. Self-service backup request Figure 9: Self-service backup request Technologies used: Veeam Backup & Replication Microsoft System Center 2012 Service Manager Microsoft System Center 2012 Orchestrator Description of the scenario: In this scenario, an end user or customer wants to create a backup of a VM. The end user requests the backup through the self-service portal of Service Manager 2012 and its request catalog. A request ticket is created and awaits approval (optional). The users line manager approves the request and an automation runbook kicks in. This runbook runs a PowerShell script against the Veeam Backup & Replication server and creates a backup job with the specifc 17 Bringing Automation to the Datacenter needs for that server and then runs a backup of that server. Finally, the end user is notifed that the request (in this case, a server backup) has succeeded and closed. Self-service virtual machine recovery Figure 10: Self-service recovery Technologies used: Veeam Backup & Replication Microsoft System Center 2012 Service Manager Microsoft System Center 2012 Orchestrator Description of the scenario: In this scenario, an end user wants to automatically restore a virtual machine. The end user browses to the self-service portal of System Center 2012 Service Manager, selects the virtual machine and indicates whether or not this restore is urgent. Optional: An approval from IT or a line-manager can be required within this workfow. The runbook starts and, depending on the urgency, an Instant or Full VM Recovery will be triggered through PowerShell. Finally, the ticket will be resolved/closed and the end user will be notifed. 18 Bringing Automation to the Datacenter Automated recovery Figure 11: Automated recovery workfow Technologies used: Veeam Management Pack for VMware Veeam Backup & Replication Microsoft System Center Service Manager Microsoft System Center Orchestrator Microsoft System Center Operations Manager Description of the scenario: In this scenario, System Center Operations Manager and Veeam Management Pack (MP) for VMware are used to monitor your VMware infrastructure. Operations Manager detects that a VM has become corrupt or is missing and creates an incident in System Center Service Manager. A runbook will start and through the use of PowerShell in the runbook, an automated recovery of that VM will be started. When this job has succeeded, the incident will be solved automatically and a change request will be created to handle the unexpected issue. Snapshot handling Figure 12: Snapshot handling 19 Bringing Automation to the Datacenter Technologies used: Veeam MP for VMware Microsoft System Center Service Manager Microsoft System Center Orchestrator Microsoft System Center Operations Manager Description of the scenario: In this scenario, System Center Operations Manager and Veeam MP are used to monitor your VMware infrastructure. Operations Manager detects that a VM has a snapshot that is overdue or has become too big. An incident is created in Service Manager and an Orchestrator Runbook is started. The runbook will remove the snapshot, update and close the incident. Virtual machine optimization Figure 13: Virtual machine optimization Technologies used: Veeam MP for VMware Microsoft System Center Service Manager Microsoft System Center Orchestrator Microsoft System Center Operations Manager Description of the scenario: In this scenario, an IT administrator reviews the VM resource allocation regularly (weekly, monthly) and, based on the information, requests a reconfguration of a VM. 20 Bringing Automation to the Datacenter Step 1: Reviewing the Right-Sizing report In System Center Operations Manager with the Veeam MP, the IT administrator runs the Right-Sizing VMs Oversized for Memory and CPU report. Figure 14: Oversized report for virtual machines Based on this information, the IT administrator requests a confguration change for a VM (DC02). Because there is an entire workfow for this job, the IT admin requests this through the System Center Service Manager self-service portal. Figure 15: Self-service portal The IT administrator flls in the requested data in the self-service portal and confrms the request. Now the request will be created as a service request in System Center Service Manager. 21 Bringing Automation to the Datacenter Figure 16: Service request information The person responsible for the service request can now see the ticket and review the user input, all the information necessary for this request and the activities that are associated with this request. Figure 17: Activities in the service request 22 Bringing Automation to the Datacenter In this case, there are two activities associated with this requesta review activity and an automated runbook activity. For this scenario, the review activity is needed in the workfow because a manager needs to approve this request. Figure 18: Review activity After that, the manager approves the request; the runbook activity starts and performs the confguration automatically. Figure 19: Runbook Figure 19 shows that specifc runbook. The frst activity in the runbook collects the user input. Based on that information, the runbook queries the VM on the VMware infrastructure with that specifc VM name. The next activity uses that information to query all the properties of that VM. You will need to stop the VM to reconfgure it and restart the VM after you have reconfgured it (note that this is carried out automatically in the next three activities). 23 Bringing Automation to the Datacenter Figure 20: Status in vCenter Figure 20 shows the status in vCenter, which you can follow while the runbook is running. Finally, when everything is fnished, the service request is closed and the end user can see the steps that were performed for his or her request. Figure 21: Updated and fnished service request ticket Conclusion Automation in a modern data environment has become a very important strategy. This white paper shows you the advantages, challenges, best practices and some examples for building automation workfows in your environment. Using the Microsoft System Center suite and Veeam technologies together allows you to enhance that automation process, so that you can build advanced workfows that include automatic backup and recovery scenarios, DR testing, automated error resolution and much more. 24 Bringing Automation to the Datacenter About the Author Mike Resseler is a Product Strategy Specialist for Veeam. Mike is focused on technologies around Hyper-V and System Center. With years of experience in the feld, he presents on many occasions at large events such as MMS, TechEd and TechDays. Mike has been awarded the MVP for System Center Cloud and Datacenter Management since 2010. His major hobby is discussing and developing solid disaster recovery scenarios. Additionally, he has enterprise-class experience in private cloud architecture and deployment, with marked focus on protection from the bottom to the top. He holds certifcations in many Microsoft Technologies such as MCITP. Follow Mike on @MikeResseler or @Veeam and on Google+. About Veeam Software Veeam is Modern Data Protection. We believe todays IT requirements have changed and that 3C legacy backup problemshigh costs, increased complexity and missing capabilitiesare no longer acceptable for any organization. Veeam provides powerful, easy-to-use and affordable solutions that are Built for Virtualization and the clouda perfect ft for the modern datacenter. Veeam Backup & Replication is VMware backup, Hyper-V backup, recovery and replication. This #1 VM Backup solution helps organizations meet RPOs and RTOs, save time, eliminate risks and dramatically reduce capital and operational costs. Veeam Backup Management Suite combines Veeam Backup & Replication and Veeam ONE in a single integrated solution to protect virtualization investments, increase administrator productivity and help mitigate daily management risks. Veeam Management Pack (MP) extends enterprise monitoring to VMware through Microsoft System Center. Veeam also provides free tools for the virtualization community. Learn more by visiting http://www.veeam.com. 25 Bringing Automation to the Datacenter Modern Data Protection Built for Virtualization Powerful Veeam Backup & Replication Easy-to-Use To learn more, visit http://www.veeam.com/backup #1 VM Backup for VMware and Hyper-V Virtualization changes everything especially backup. If youve virtualized on VMware or Hyper-V, now is the time to move up to the data protection solution Built for Virtualization: Veeam Backup & Replication. Unlike traditional backup that suers from the 3C problem (missing capabilities, complexity and cost), Veeam is: Join the 58,000 organizations who have already modernized their data protection with Veeam. Download Veeam Backup & Replication today! Powerful: Restore an entire virtual machine (VM) or an individual le, email or database record in 2 minutes Easy-to-Use: It just works! Aordable: No agents to license or maintain, works with your existing storage, and includes deduplication, VM replication, Microsoft Exchange recovery, and more!
GOLD AWARD NEW TECHNOLOGY GOLD AWARD NEW TECHNOLOGY