A Day in Life of Datawarehouse Architect part 1

A data warehouse Architect generally help to design datawarehouse , requirement gathering in ETL Low level design LLD, and HLD high level design, setting up database infrastructure design for datawarehouse like Storage Area Network requirements, Rapid application Clusters for database of datawarehouse more details read
Datawarehousing consists of three main area :
1. ETL(data migration, data cleansing, data scrubbing, data loading )
2. Datawarehouse design
3. Business Intelligence (BI) Reporting infrastructure.
BI
Read These Two part article for BI
– https://sandyclassic.wordpress.com/2014/01/26/a-day-in-life-of-bi-engineer-part-2/
– https://sandyclassic.wordpress.com/2014/01/26/a-day-in-life-of-business-intelligence-engineer/
And Architect
https://sandyclassic.wordpress.com/2014/02/02/a-day-in-life-of-business-intelligence-bi-architect-part-1/

Design : Now Coming to part 2 (is generally work of Data warehouse architect)
Read Some details More would be covered in future articles
https://sandyclassic.wordpress.com/2013/07/02/data-warehousing-business-intelligence-and-cloud-computing/
9:00-9:30 Read and reply mails.
9:30-10:30 Scrum Meeting
10:30-11:30 update documents According to Scrum meeting like burn down chart etc..update all stake holders.
11:30-12:00 Meeting with Client to understand new requirements. create/update design specification from requirement gathered.
12:00-13:30 create HLD/LLD from the required user stories according to customer Landscape of technology used.
13:30-14:00 Lunch Break.
14:00-14:30 Update the Estimations ,coding standards , best practises for project.
14:30-15-30 Code walk through update team on coding standards.
15-30-16:30Defect call with Testing and development Team to understand defects, reasons of defects, scope creep, defect issuse with defect manager, look at issue/defect register
16:30-17:30 Work on specification of Design of datawarehouse modelling Star or Snow flake schema design according to business requirements granularity requirements.
17:30-18:30 Look at Technical Challenges requiring Out of Box thinking, thought leadership issue, Proof of concept of leading Edge and Breeding Edge technologies fitment from project prospective.
18:30-19:30  onwards Code for POC and Look a ways of tweaking , achieving technology POC code.
19:30- 20:30 onwards Forward thinking issue might be faced ahead by using a particular technology is continuous never ending process as there can be multiple combination possible to achieve as well as using particular component or technology should not create vendor lock in, cost issues, make/buy cost decisions, usability, scalability, security issues (like PL/SQL injection, SQL injection using AJAX or web services may be affected by (XSS attack or web services Schema poisoning), Environmental network scalability issues. Affect due to new upcoming technology on Existing code.
20:30 Dinner
Available on Call.. for any deployment, production emergency problems.

A day in Life of datawarehousing Engineer Part-2

Read Previous part
https://sandyclassic.wordpress.com/2014/02/19/a-day-in-life-of-datawarehousing-engineer/
Normal Schedule for development role :
9:00-9:30 Check all mail communications of late night loads Etc.
9:30-10:30 Attend Scrum meeting to discuss update status of completed task mappings and mapping for New user stories requirements, understand big picture of work completed by other staff status.
10:30 am -1:30 pm Look at LLD, HLD to create source to target transformations after understanding business logic and coding that in transformations available with tool.
1:30-2:00 Lunch break
2:00-3:00 Unit test data set to validate as required between source and target.
3:00-3:30 Documentation requirements of completed work.
3:30-4:30 Attend defect Call To look into new defects in code and convey back if defects not acceptable as out of scope or not according to specifications.
4:30-5:00 Status update daily work to Team Lead.
5:00-5:30 sit with Team lead, architect code walk through and update with team.
5:30-6:30 Take up any defects raised in Defect meting and Code walk through.

A day in Life of datawarehousing Engineer Part-1

Datawarehousing consists of three main area :
1. ETL(data migration, data cleansing, data scrubbing, data loading )
2. Datawarehouse design
3. Business Intelligence (BI) Reporting infrastructure.
BI
Read These Two part article for BI
https://sandyclassic.wordpress.com/2014/01/26/a-day-in-life-of-bi-engineer-part-2/
– https://sandyclassic.wordpress.com/2014/01/26/a-day-in-life-of-business-intelligence-engineer/
And Architect
https://sandyclassic.wordpress.com/2014/02/02/a-day-in-life-of-business-intelligence-bi-architect-part-1/
Design : Now Coming to part 2 (is generally work of Datawarehouse architect)
Read Some details More would be covered in future articles
https://sandyclassic.wordpress.com/2013/07/02/data-warehousing-business-intelligence-and-cloud-computing/
Part 1: ETL Engineer:
Most common task of ETL (Extract data- Transform data -Load to target).
Most Common ETL Tool being
Independent Tool: Informatica, IBM data stage, Ab-initio, Terradata ETL utilities,
Tool within ERP: SAP BIW ABAP based transformations , LSMW, peoplesoft EPM (internally uses other tools though).
Tool within Databases: Oracle SQL loader, Teradata ETL utilities,(Tricle pump, multi-load, fast load),
Microsoft BI Stack with SQL server had : SSIS SQL server integration services.
Cloud based Tool: Apache Hadoop Hive datawarehouse ( here requirement is different from un-structured realtime data analysis.
First data modelling have to completed to have level of granularity to represent requirements of business key drivers.
Once datawarehouse Structure in completed to ascertain level of granularity required.
The data loading Cycle Starts With:
Extraction from desperate data sources in Stagging area
On Stagging area data is cleansed.
Then data Transformation are applied for
Example in Informatica and SSIS:
https://sandyclassic.wordpress.com/2014/01/15/eaten-tv-from-partly-eaten-apple-part-2-artificial-intelligence/
Two Sets of documents are There LLD and HLD to look at what needs transformation to be applied.
Like in Informatica Transformation Types are :

Informatica Transformation Types


Look at all transformations available in Informatica version 9
http://www.folkstalk.com/2011/12/transformations-in-informatica-9.html
These can be customized according to logic required.
Next step is Loading to datawarehouse dimension tables  and then to Fact table.
Read: https://sandyclassic.wordpress.com/2014/02/06/coke-vs-pepsi-of-datawarehousing-etl-vs-elt/
And more
https://sandyclassic.wordpress.com/2013/07/02/data-warehousing-business-intelligence-and-cloud-computing/

 

Case Study Artificial Intelligence,ETL and Datawarehousing Examples part 1

Read : IPTV and Augmented Reality using Artificial Intelligence.
https://sandyclassic.wordpress.com/2012/06/27/future-of-flex-flash-gamification-of-erp-enterprise-software-augmented-reality-on-mobile-apps-iptv/

AI is there in many place like one area of AI Fuzzy Set there is already Fuzzy Transformation in SQL Server Integration Services since year 2010.
What it does Fuzzy logic Transformation achieve?
So when we match two records we do it by checking each alphabet using regular matches.
But when we use fuzzy logic it brings out similar sounding and combination matches although alphabet may not be same also it checks meaning is same. Even it can override spelling mistakes to get right results How?
Example Fuzzy logic in SSIS:
USA,us, united states – For country Any person can enter any of these combination.
Usually its taken up for Data cleansing.
If data is not cleaned using De-dup it may not show many of these records in result for matches.
But Fuzzy logic we use Fuzzy set from all records it creates fuzzy set of record with
Set A { ElementA, membershipOfElementA}
membershipOfElementA define in percentage terms the possibility of it being in the similarly grouped set.
{us,0.97} {united states,0.98} {usa,0.99} {united states of America,1} so we can set tolerance level to 3% then all of these matches are there in result.
code you can see at http://www.codeproject.com/Tips/528243/SSIS-Fuzzy-lookup-for-cleaning-dirty-data
SIRI:  Speech Recognition Search Which was introduced in iPhone long back takes speech.
Speech input to pressure sensor –> generate Waveform –> Then Compare wave form
That’s process but.
AI in SIRI
The Waveform may be amplitude modulated but yet same thing let suppose we say
Apple the Two Waveform compared may have boundary level aberrations which can be defined by membership function Then same result within same Tolerance limit can be deemed to be similar. This membership can be calculated each time person do a search dynamically when it says something in on Mike which repeat same process again.
There can be lots of image processing and AI search algorithm can be built to make better.
Like A* search etc.
Already if the words are linked can be understood by Neural Network. Similar way Neural Network is used to predict The  traffic congestion aggregating data paths from street light sensors in japan Tokyo.
Aggregation of words can be achieved by neural Network in not exact but similar way to some Extent. Thus completing the search.
This aggregation may be used in text, covariance matrix of images or covariance of sound score or speech search.
Using Laplace Transform’s Cross correlation. Read (http://en.wikipedia.org/wiki/Cross-correlation512px-Comparison_convolution_correlation.svg
Now TV is large platform just like difference between watching movie on laptop or TV Vs on 70 mm screen. Each of those has there own market.
Costly Miniaturisation
What effect you can provide on TV may not be provided on mobile until there is technical break through in miniaturisation. I am not saying it cannot be provided but it will require relative less technical  break through compared with miniaturized chips or may be less costly.
Second TV is like we have last mile connectivity in Telecom.
So When you have something to watch in any storage device you can just throw that on TV Ubiquitously . As TV would be there in every house so you need not carry screen to watch. Just like Last mile wireless connectivity using HotSpot.

Economic Development and National integration of North East India

Read Previous Blog entries problems in India National Integrations and solutions:
1. https://sandyclassic.wordpress.com/2014/02/03/world-moving-towards-reconciliation-india-moving-backwards/
2. https://sandyclassic.wordpress.com/2014/01/02/what-can-bring-fast-development-in-biharupwest-bengal/
3. https://sandyclassic.wordpress.com/2014/01/30/india-problems-in-national-integration-and-solutions/
North East of India has unique culture with Each of 7 states have unique classical dance forms, Folk, Mighty Himalayas, common Brahmputra, and its own Martial Art forms.
Manipur:
a. Watch these two beautiful dance forms devoted to Krishna as aerobatic as Brazil martial dance forms.

b. Manipur dance on life Krishna raslila with Radha
http://en.wikipedia.org/wiki/Manipuri_dance
Similarly other parts of North East Assam has Bihu, Nagaland very aerobic dance..
Toursim spots.
The Economic Success of North East lie with linking it to Myanmar, Thailand, Malaysia , Cambodia and Singapore with high speed Road and Rail connectivity.
Like in Ancient times it can become trade route between India and East Asia, China and Bangladesh As more people travel through the region people will be able to understand the diversity of India.
How Linking helps North East, and Bengal, Bihar, UP? Read above article and below one
https://sandyclassic.wordpress.com/2013/11/13/pathetic-state-of-economic-project-execution-ways-to-take-forward/
High Speed Rail Economics:
https://sandyclassic.wordpress.com/2013/12/30/travel-bombay-to-delhi-in-5-hrs-high-speed-train-save-140-billion-dollar/

2. Teaching North East Dance forms and Martial Art forms to Rest of India.
North East dance form are very aerobatic can be used for people to learn art as well workout to reduce obesity. In directly It will show people in rest on India cultural awareness about North East.
There is huge market to learning in rest of India let say: Thang-Ta
1. http://www.youtube.com/watch?v=AB2G1kTD-SY
2. http://www.youtube.com/watch?v=f3Uh6BI2VuE
Government and proviate sector can pitch in create these kind of course in rest of India like for obessity reduction with dance can popularise this dance form

Similarly Naga dance, Arunachal, Each Region has so many skills to teach in schools in India. Really it makes me surprise Why its not taught in schools in India?
Assam Bihu, Arunachal pradesh Folk dance each part of 7 states Has so many music and dance forms to Offer but yet not taught in Any School in India..