Building BI Environments on a Budget

converseoncologistInternet and Web Development

Aug 7, 2012 (4 years and 11 months ago)

307 views

Building BI Environments on a
Budget

Eric Vallo

BI Solutions Architect

My Point of Reference


Formerly a BI architect for two large
enterprise customers


Now a BI architect for a company with less
than 2,000 employees


Successfully completed a Crystal Enterprise
10/Business Objects 6.5 migration to XI R2


Now challenged with big requirements but
limited financial resources to implement a
Business Objects environment

Introduction


Constructed in a robust, Services Oriented
Architecture (SOA), Business Objects
provides endless configuration options to
satisfy business requirements


Budget constraints are always a
consideration when building out an
environment, but don’t necessarily have to
be a limiting factor

Project Scope vs. Budget


In past lives, big BI projects typically came
with big budgets


Working with a smaller customer, smaller
project sizes result in dramatically smaller
budgets


The outcome ultimately has to be the same,
but the approach is entirely different



Typical Requirements


Satisfy all the reporting requirements


Provide appropriate performance levels in
accordance with service level agreements


Have some type of redundancy built into the
architecture


Ensure that the environment can scale with
the growth of the reporting needs



Deconstructing the Layers of
the Architecture

High Level Architecture


Start by thinking of the BO
architecture in a very
abstract form


Simply put, your
architecture can be sliced
into distinct layers


Each layer comes with its
own capabilities to achieve
your organizations goals
for capability and
availability


Presentation Layer

BI Layer

File Repository

Database

Repository

Presentation Layer


The Presentation Layer is physically
identified by the web server environment that
renders the final content to report consumers


Handles all of the HTTP traffic to and from
the report consumers


Also accessible via external applications
doing SDK development


Typical deployments are on .NET (IIS) or
Java (Tomcat, WebLogic, WebSphere, etc)



BI Layer


The BI Layer is where Business Objects
actually resides


Handles communication between all layers
and services


Manages security and content



File Repository


The File Repository is where Business
Objects content is physically stored


Structure is a file system, organized in
folders based on unique IDs in the Business
Objects system



Database Repository


The Database Repository, known as the
Central Management Server database


The metadata database that defines all
objects, security, etc.



Constructing the
Environment

A Case Study


Very small deployment, currently one server
running two CPUs


Small user base less than 1,000 users


Java web application shop


Incrementally design an environment that
provides fault tolerance and can still scale
beyond 1,000 users



Presentation Layer Design


Deploy the Presentation Layer on physically
separate hardware from the BI Layer, File
Repository, and Database Layer


Great candidate for Linux based servers


Great candidate for VMware


Easily load balanced using industry standard
hardware


Doesn’t incur additional Business Objects
license expenses



Load Balancing Deep Dive


Assumption: Most organizations
today employ some type of
hardware or software load balancing
technology


Prior knowledge with Load
Balancers is not necessarily
required to include in architecture


Generally architected with primary
and redundant load balancers


Creates a virtual IP address for your
clients to access


Make friends with your Network
Engineer, because he/she holds the
keys to this capability



Load Balancing Deep Dive


Leverage “sticky” sessions


Lower the polling for failed
servers


Lower the retries to detect a
server failure


Successfully proven to
minimize impact to online
users in the event there is a
node failure


Enables you to easily add web
servers if the load is too great
across your existing servers


This is what makes scaling
with VMWare attractive!




VMWare Deep Dive




Constructed correctly, VMWare can provide the ability to rapidly scale an
environment


Clustered virtual infrastructure allows for a VM to run on any
hardware in the physical cluster


Resource scheduling evaluates load real time and moves VMs to
server with least load


grid computing concepts


Fault tolerance


if the physical server fails, the time to migrate the
server state is approximately 12 seconds


Physical server maintenance does not impact the VM in a clustered
environment


Ease of upgrades to allocate CPU/Disk/Memory to a VM


Hot backups can be performed to restore server state


Additional VMs can literally be added within an hour

BI Layer Design


Again, deploy on physically separate
hardware from the other layers


Typically deployed on Unix based or Wintel
architecture


Great candidate to evaluate on Linux! I am!


Most important part of the architecture to
ensure is on physical hardware


Load balancing already built into Business
Objects




BI Layer Redundancy

BO licenses aren’t cheap.

How do I get redundancy?


Solution:

Active/Passive Configuration




BI Layer Design


Hardware is cheap, so get an
identical server to the primary
Business Objects server for
redundancy


Consider Linux for a lower cost
environment configuration if
necessary skill sets are present


Install it just like it were part of
the cluster and test it


When 100% operational, stop all
services on the passive server


Utilize 3
rd

party monitoring such
as OpenView to monitor system
and fail over when required




Making it Fail Over “Automagically”


Health check examples might
include CPU, Memory, or Disk
utilization, or even checking the
availability of your server’s logon
page


If failure is detected, invoke script to
enable services on your passive
server


Use the monitoring tool to throw an
alert that your environment has
been failed over via email/pager


Minimizes the impact to users on
the platform while you fail over to
the passive server and keeps your
operation running



Leverage Multi
-
Core Architecture


Assume any new server will include multi
-
core technology,
resulting in a relatively new approach to licensing


Secondary cores will typically deliver between 50
-
75% of
the performance of the primary core


Resulting CPU licensing is 50% the cost of a full CPU license per
second core


2 CPU x 2 Cores Each = 3 CPU license requirement


Look for hardware that allows you to disable the second
core on each chip This can effectively reduce the number of
CPUs purchased up front to two


Enables an incremental CPU increase on a server of one CPU
before additional servers/licenses may be required


2 CPU x 1 Core Each = 2 CPU license requirement


Same concept holds true whether purchasing a two, four, or even an
eight CPU server



Strategies in Licensing


Consider named user licensing vs. CPU based
licensing for the initial environment construction


Ensure any master agreement in place is flexible
enough to permit conversion of named to CPU
licenses


Remember to profile users for creators vs.
consumers, so you can differentiate licensing
between Business Objects Enterprise and Web
Intelligence licenses


Where To Next?


Proactively monitor your server performance down to the
service level (CMS, Tomcat, Web Intelligence, etc.)


If capacity is available on existing components of the
cluster, scale up first before scaling out


Stretch the limits from the default BO settings to allocate additional
resources to existing processes


Add additional job servers


Add additional report servers


As projects roll in, incrementally increase spending to
augment the environment to upgrade servers, add more
servers, add BOBJ licenses, etc.


The architecture picture painted so far means that you can
gradually add components to each layer to meet your goals



File Repository Server Design


Again, deploy on physically separate
hardware from the other layers


Can be deployed on pretty much any
hardware that has a file system which can be
recognized by your Business Objects servers


Another great candidate for VMWare


Great opportunity to piggy back on any
Enterprise storage solution




Enterprise Storage


This is the right time to make
friends with your Storage
Manager


Ensure your file server has the
right connectivity, typically
fiber, to hook into a SAN
solution


This provides higher availability of
your file system


The File Repository Server <>
a huge server


It just needs to be able to handle
the I/O to the SAN solution



VMWare Revisited


High availability through rapid replication in a
clustered environment


Illustrates grid computing concepts


Rapid expansion capabilities


High probability that it can avoid outage
impacts altogether with its failover
capabilities




Database Repository Design


Again, deploy on physically separate
hardware from the other layers


A vast array of database
technologies are supported here


Pick one that your organization has
expertise with


Be aware that database high
availability solutions can be
extremely expensive


The best strategy on a tight budget:


Piggy back on an existing database
environment


Ensure a strong backup and recovery
strategy




Bonus


External Access


Through the use of a DMZ/Proxy,
external traffic can easily leverage
the same architecture


The proxy handles encryption and
transmission to external clients


If external users are non
-
Active
Directory accounts, LDAP or BOE
security can easily be leveraged here
as well


Whether you open up your BO
environment externally all depends
on the needs of the business


Also consider Citrix for external
access to the platform



Recommendations

and

Conclusions

Recommended Approach

1.
Understand all Enterprise Resources that are available
(Load Balancing, VMWare, Enterprise Storage,
Databases)

2.
Get control of the BI Layer


Scale up before you scale out, if possible


Add a second server to the cluster
-

By its nature, requires you to
implement a shared file system for your File Repository Server

3.
Redistribute the HTTP traffic to stand alone servers


Quickest bang for the buck by moving services such as Tomcat to
a stand alone server

4.
Evaluate if a high availability database instance is present
in your organization in order to leverage its strengths


Bottom line: a small environment can exhibit all the
strengths in stability as a large scale environment






Conclusions


The environment can scale up or out


Scale up has already occurred for Crystal Reports Job Server and
Web Intelligence Report Server


We have increased to this configuration in response to
increased demand in both users and scheduled reporting


For my organization, this environment currently supports


200+ recurring Crystal Reports


600 Web Intelligence users


Early stages of SharePoint integration


Early stages of integration with WebSphere Portal Server


Live Office and Query as a Web Service coming soon


Uses Active Directory for Internal employees


LDAP authentication coming soon for external access




Questions?

Eric Vallo

BI Solutions Architect

eric.vallo@gmail.com