Both Provider and Customer will appoint a service level account manager to be the central point of contact for all service levels (“SLA”), general account information and management.
This SLA will be reviewed on an annual basis between Provider and Customer.
(a) Minimum Up Time. Provider shall eliminate any Downtime or intermittent order processing issues within its control, and shall provide a minimum of 99.9% Up Time (as defined in the following subsection) each month.
(b) Calculation of Up Time. Up time (“Up Time”) is to be measured by (i) the total number of minutes during a calendar month in which the Services are completely available and usable, excluding time spent on Approved Maintenance, (ii) divided by the total number of minutes during the calendar month. “Approved Maintenance” shall mean notified and scheduled maintenance performed during scheduled maintenance windows.
The following formula shall be used to calculate Up Time for a given month:
Up Time = (Total minutes in the calendar month – Total minutes Downtime in such month) / (Total minutes in the calendar month)
“Downtime” shall mean (i) any lapse in availability of the Services, calculated from the service interruption and ending when the service is restored, provided the outage occurred within the Provider facility or was otherwise within Provider’s control. Downtime shall not include any lapse in availability due to Approved Maintenance.
(c) Payments by Provider for Failure to Maintain Minimum Up Time. In any month in which Provider fails to maintain Up Time of 99.9%, then Provider shall grant Customer a credit against amounts owing to Provider in the following amounts:
The credit will be applied to the following calendar quarter’s invoice. For clarity, in the event Services are pre-paid then, in lieu of any credit hereunder, Provider shall refund the amount of the applicable credit to Customer within fifteen (15) days of the end of the month in which the credit accrued.
(d) Analysis of Failure to Maintain Minimum Up Time; Remedy Plan. If Provider fails to maintain the minimum Up Time required herein, then Provider shall within ten (10) days of notice from Customer perform a root cause analysis to identify the cause of such failure, provide Customer with a remedy plan and implement such plan in an agreed upon time frame.
(a) Maintenance. All scheduled maintenance requiring downtime will be performed between the hours of 8:00 a.m. and 12:00 pm. Eastern Time on a Saturday (“Maintenance Windows”) and require less than two hours of downtime. Provider shall provide written notification to Customer at least 3 days in advance of such maintenance. Any major maintenance or upgrades that will impact the availability of the Services outside the hours of 8:00 a.m. to 12:00 p.m. Eastern time on a Saturday must be discussed with Customer in advance and the parties shall mutually agree to a time that will have the least impact on the availability of the Services.
(d) Unscheduled Downtime. In the event of a system failure, Provider will “fail-over” to an alternate system within 15 minutes, and shall bring up the entire infrastructure and the Services from a cold boot in a maximum of two hours. In the event of such a system failure, Provider will notify Customer of the unscheduled downtime and the status of the event via a telephone conversation. Provider will escalate the issue within Customer until live human contact is made.
(e) Catastrophic Failure. In the event of multiple catastrophic system failures, Provider shall find a working solution or switch to a backup server in a maximum of forty-eight hours. In the event of such a catastrophic system failure, Provider will notify Customer of the unscheduled downtime and the status of the event via a telephone conversation. Provider will escalate the issue within Customer until live human contact is made.
(a) Data.
(i) Provider shall maintain all data on a reasonable amount of equipment necessary to meet Up Time and maintenance requirements.
(ii) If any non-database data is lost then it shall be copied back from backups, alternate systems, or website/code repositories.
(iii) Provider shall perform nightly backups of all data and information generated, collected, handled, stored, input and processed by the Services activity. Backups will be stored or copied to a secure offsite storage facility.
(iv) Non-database data that is no longer in use will be removed and stored for 6 months to a year.
(v) Provider shall test database restores quarterly.
(b) Recovery Plan. If a full system restore is required, Provider will provide services on an alternate system(s), if necessary, while such restore is in progress. Integration of the restored system into production will occur at a time mutually agreed upon by Customer and Provider.
(c) Audit Trails. Provider shall maintain weekly access logs to its servers and shall maintain all access logs on backup for at least one year.
(a) Internet Service Providers. Provider shall maintain at least two Internet Service Providers (“ISPs”) with multiple paths into the building. If any of the ISPs have difficulties, traffic shall automatically be rerouted to another carrier.
(b) Fire. Provider’s Service facilities shall contain an automated fire suppression system that will not affect any of the equipment or systems but will immediately extinguish a fire. The Service facilities shall also contain individual fire extinguishers that are safe to the systems and equipment.
(c) Bandwidth. In addition to the two ISPs and the automatic fail over between ISPs, Provider shall maintain two identical routers and an alternate firewall server. The back-up router and Firewall shall be pre-configured and able to be brought online immediately.
(d) Power. Provider’s Service facilities shall have multiple sources of power including heavy-duty utility feed, extensive battery backup and a generator. All systems shall be supported by an automatic transfer switch that will switch all power requirements to battery and start the generator in the event of a power failure. The generator shall have sufficient capacity to allow Provider to continue to meet its obligations hereunder (including customer service and development, if applicable) until power is restored. Provider‘s production facility shall not have any single point of failure with electrical power.
(e) Server Failure. Provider’s system shall be redundant to a reasonable degree necessary to meet Up Time and Maintenance requirements.
(a) Traffic. Provider shall utilize a monitoring program to track its bandwidth and the status of individual data lines. In addition, Provider shall capture such data required for reporting purposes.
(b) Performance. Provider shall use tool(s) for monitoring URL and database response, and built-in database performance monitors.
(a) Organization. Provider shall maintain a dedicated staff of at least two support engineers. Provider shall have a mechanism to contact member(s) of its development team from 8:00am to 8:00pm ET to respond to critical or serious errors that require programming or other developmental support.
(b) Escalation Plan.
(i) Problem Reporting.
(1) Provider shall maintain monitoring tools that report all problems and contact support engineers by one or more methods as is appropriate to the level of the problem. Provider’s Service facilities onsite staff shall be notified when an error is discovered. Such staff shall respond on a priority basis. Provider shall maintain "on-call" and "on-call backup" administrators available 8am-8pm ET, and Provider’s or a subcontractor’s Data Center’s staff and senior management shall monitor their progress.
(2) After making the appropriate corrective action, the administrator shall submit resolution information to the Provider SLA account manager if the problem affected Customer.
(ii) Resolving Problem Reports.
(1) Provider’s monitoring programs shall allow for proactive response to avoid problems and alert system administration team members who will escalate as is appropriate.
(2) In the event of any third party-detected problem, Provider’s customer service department shall contact the Provider SLA account manager. Customer Service team members shall also be trained to monitor basic system functions throughout their shift, and report any issues to the Provider SLA account manager.
(3) Once the Provider SLA account manager is contacted, such individual(s) shall take steps to implement a solution as is appropriate to the level of the problem. Any problem that causes or can cause a greater than 20 minute down time shall immediately be escalated to senior management.
(4) Provider shall provide all downtime reports and/or incident reports to Customer. Provider will permit Customer to be involved in any part of the alert process upon its request.
(a) Personal and Financial Information.
(i) Transmission. All critical information shall be encrypted using SSL in the http interface and DES or equivalent or better encryption, using the TCP/IP API interfaces.
(ii) Storage. Data shall not be accessible directly from the web server or directly from the database. All data must all go through proper page or API requests, which are abstracted from the data, firewall, database and web server security block access to any data.
(b) System Integrity.
(i) Physical. All visitors shall be escorted by an authorized employee of Provider. Facility breach alarms may automatically generate local police notification, subject to Provider’s alarm response procedures and the area or zone where the alarm was triggered.
(ii) Operational.
(1) Administrative access shall be controlled by a security login system, which assigns valid day, time of day and functional access. Such system shall also track and document access to the system as is reasonable to protect Customer partner websites and data.
(2) Provider shall utilize industry-leading software and industry standard database techniques.
Customer shall have the right to acquire Support for all Provider Software on an annual basis. Support shall include at a minimum the right for Customer to receive (and an obligation for the Company to provide) all of the following components:
A. Error corrections, workarounds, bug fixes, patches, modifications, enhancements and other updates which are released to the Company’s other supported customers. Upon request, the Company shall provide to Customer a listing of all available updates currently available to the Company’s supported customers. The Company shall deliver to Customer each update as soon as it is generally available in production release;
B. An email account, which shall be staffed twelve (12) hours per day and five business days per week throughout the Support period, by individuals skilled in Provider Software technical support matters;
D. Email notification services for general Support issues.
Initial-year Support for a particular Provider Software license shall commence on the applicable Acceptance Date for such Provider Software license and shall continue until for one (1) year thereafter. Support renewal terms shall commence on the effective date thereof and continue for a one (1) year period thereafter.
If on-site assistance is required and the reported error is due to a failure of the Provider Software to materially perform to the applicable specifications, the Company shall provide on-site assistance upon request by Customer, at no further charge. If the failure has conclusively been determined to have resulted from Customer’s actions, Customer shall pay the Company consultant’s hourly rate. Should Customer allow the Company to directly or remotely access Customer’s network(s) during the course of performance of any Support, the Company shall first execute Customer’s then-current standard network access agreement or other agreement of a similar nature.
For supported Provider Software licenses, Customer shall have the right to contact the Company’s Support organization in accordance with the procedures specified herein. Initial calls into the Company’s Support organization will be: (i) serviced by a Support employee within 1 hour or placed directly into an expedited Support queue, (ii) logged into the Company’s call tracking system, and (iii) assigned a case number. Customer shall classify each Support request in accordance with the Severity levels specified below and the Company shall take the following actions as specified below. All responses shall at a minimum be provided by the Company in English and/or any other language specified by Customer to the extent that skilled Support staffing is available with the requested language capabilities.
Definition: A problem has been identified that makes the continued use of one or more functions impossible (or severely restricted) on a critical system and prevents continued production or severely risks critical business operations. Problem may cause loss of data and/or restrict data availability and/or cause significant financial impact to Customer.
Response Commitment: A Support engineer will respond to a call within one (1) hour of a problem report. The Company will promptly assign Company Support engineers to investigate the problem report and notify the Company a Severity 1 Error has been reported and of the steps being taken to correct such error. The Company shall provide Customer with at a minimum reports every three (3) hours on the status of the investigation. The Company shall initiate work to provide Customer with a workaround or fix. If a workaround or fix is not available within four (4) hours, the problem report shall be escalated to the Company’s product development organization. The Company and its personnel shall provide uninterrupted continuous effort on a 24X7, 7X7 basis until a Severity 1 Error is corrected or a workaround acceptable to Customer is implemented.
Definition: A problem has been identified that severely affects or restricts major functionality. The problem is of a time sensitive nature and important to long-term productivity but is not causing an immediate work stoppage. No workaround is available and operation can continue in a restricted fashion.
Response Commitment: A Support engineer will respond to a call within two (2) hours of a problem report. The Company will promptly assign Company Support engineers to investigate the problem report. The Company shall initiate work to provide Customer with a workaround or fix. If a workaround or fix is not available within twenty-four (24) hours, the problem report shall be escalated to Company Development.
Definition: A minor problem that does not have major effect on business operations or a major problem for which a Customer acceptable Workaround exists.
Response Commitment: A Support engineer will respond to a Customer call within two (2) days of a problem report. The Company will promptly assign Company Support engineers to investigate the problem report. The Company shall initiate work to provide Customer with a workaround or fix. If a workaround or fix is not available, the Company may include a fix for the error in a future Releases.
Customer shall have the right, but shall not be obligated, to renew Support pursuant to an Order Form. The Company shall notify Customer in writing of any renewal pricing for renewal year Support Services at least sixty (60) days prior to the expiration of the applicable Support period by providing a quote for renewal year Support prior to the 60-day period. In the event that the Company fails to notify Customer of any renewal pricing within the time provided, the Company shall provide Customer with renewal year Support at no further charge to Customer.
In the event of any lapse in Support Services coverage, the Company shall allow Customer to renew Support without any additional charge other than the payment of the applicable annual Support fees mutually agreed upon by the parties for the renewal Support period. Customer may cancel ongoing any Support acquired by Customer at any time by providing ten (10) days prior written notice of cancellation. Upon any such cancellation, the Company shall refund to Customer any remaining, unused prepaid Support fees for the applicable Support period cancelled, pro-rated on a monthly basis.